Ready reckoner
Threat modeling
| Method | Use when |
|---|---|
| STRIDE | Systematic per-component/flow coverage |
| PASTA | Tie threats to business impact (7 stages) |
| MAESTRO | Agentic/multi-agent; threats within and between 7 layers |
- Cover the whole pipeline; use libraries: MITRE ATLAS, OWASP LLM/agentic Top 10, NIST AI 100-2.
- Actors by motivation, capability, access; include non-human actors (your own manipulated agent).
- Classic families: evasion (inference, integrity) · poisoning/backdoor (training) · extraction (confidentiality) · inversion/membership inference (privacy).
- OWASP LLM 2025: injection, sensitive disclosure, supply chain, poisoning, improper output handling, excessive agency, system prompt leakage, vector/embedding weaknesses, misinformation, unbounded consumption.
- Credible robustness claim = stated threat model + adaptive attacks + results across attack strengths.
Secure AIDLC and change
- Security activity and automated gate at plan, data, build, evaluate, deploy, operate, retire. NIST SP 800-218A (SSDF for GenAI).
- Environments: research (no prod creds) · dev · training (isolated, no internet) · prod (pipeline-only).
- Behavior changes with model, data, retrieval, prompts, guardrails, tools → all versioned in a release manifest.
- Rollout: shadow (users see old) → canary (some users see new) → pre-agreed triggers → whole-bundle rollback.
Supply chain and provenance
- AI supplier risks: training data, silent changes, data use, dependency, opaque safety tuning, sub-processors.
- Assurance: certificate scope, SOC 2 period/exceptions/user entity controls, test scope, model card method.
- Licensing applies along the pedigree (base model + data terms).
- Hash = integrity · signature = publisher · attestation = how it was built (SLSA).
- AIBOM = component present · VEX = exploitable? (not affected / affected / fixed / under investigation).
Red teaming
- Scope: objectives, scenarios, in/out of scope, measurable success criteria, signed RoE (data handling, safety stops, escalation).
- Sandboxes and mock tools for agents.
- Report ASR together with false refusal rate, over many attempts and variants.
- Human creativity + automation (PyRIT, garak) in break-fix cycles; map to ATLAS/OWASP.
- AI testing AI: keep in scope, validate the judge, protect generated data.
- Jailbreak root causes: competing objectives, mismatched generalization. Jailbreak = model breaks its own rules; injection = attacker's instructions replace the application's.
Findings
- Triage: severity, exploitability, reproducibility, business impact, reach, detectability.
- Classify: security vulnerability · model limitation · safety issue · fairness issue → right owner.
- Map to ATLAS, OWASP, CWE.
- Reproduction package: versions, settings, inputs (all turns), outputs, success rate over N, trace IDs; protected.
- Audience: engineers (repro), leaders (decision, no payloads), legal (obligations), vendors (responsible disclosure).
Mitigations
- Target the mechanism, not the symptom (blocklists chase phrasing).
- Tune for missed attacks and false positives, by risk tier.
- Prioritize by risk reduction, feasibility, business context; interim controls; recorded, time-limited acceptance.
- Regression suite (variants + utility, run N times) as a release gate.
- Residual risk: KRIs, scheduled tests, canaries, reassessment triggers.
- Empirical robustness (practical, adaptive attacks) vs certified (provable, narrow bounds). Guardrail models are one layer.