Ready reckoner
Threat modelling
| STRIDE | AI example |
|---|---|
| Spoofing | Fake agent / MCP server, voice deepfake |
| Tampering | Poisoned data, edited weights or prompts |
| Repudiation | Unattributable agent actions |
| Info disclosure | Injection exfil, data extraction, prompt leakage |
| DoS | Token exhaustion, agent loops |
| EoP | Injection drives tools beyond user's rights |
- PASTA: 7 stages, risk-centric, attack simulation — for critical systems.
- MAESTRO (CSA): 7 layers — foundation models · data ops · agent frameworks · deployment · evaluation & observability · security & compliance · agent ecosystem. Watch cross-layer threats.
- Actors: criminals · nation states · competitors · hacktivists · abusers · malicious & negligent insiders · non-human insiders (manipulated agents).
- Intel: MITRE ATLAS, OWASP, incident databases, provider advisories.
Adversarial ML
| Attack | When | Goal |
|---|---|---|
| Evasion | Inference | Wrong decision on crafted input |
| Poisoning / clean-label | Training | Degrade or target |
| Backdoor | Training / distribution | Trigger behaviour |
| Extraction | Inference | Copy the model |
| Inversion | Inference | Reconstruct training data |
| Membership inference | Inference | Was X in training? |
| Data extraction | Inference | Regurgitate PII/secrets |
| Availability | Inference | Sponge inputs, token exhaustion, denial of wallet |
White/grey/black-box; transferability via surrogates. Defences: adversarial training, DP, output minimisation, rate limits, caps, multiple signals.
GenAI threats
- Direct injection (user) · indirect injection (content the system reads) · jailbreak (bypass safety = content). Injection = control.
- OWASP LLM Top 10 (2025): 01 injection · 02 sensitive disclosure · 03 supply chain · 04 data & model poisoning · 05 improper output handling · 06 excessive agency · 07 system prompt leakage · 08 vector & embedding weaknesses · 09 misinformation · 10 unbounded consumption.
- RAG: poisoning, ranking manipulation, ACL bypass, cross-tenant leakage, embedding inversion.
- Agents: tool misuse, memory poisoning, goal manipulation, confused deputy, spoofing, cascades, oversight flooding, covert exfil (markdown image URLs).
- Multimodal: text in images, hidden audio, white-on-white PDF text, deepfakes.
Red teaming
Scope: system + pinned versions · objectives · actors · in/out of scope · rules of engagement · reporting · retest. Loop: recon → baseline → manual → automated → chain → report → retest. Report success rates (e.g. 14/50). Test whole system, every entry point; canaries; controlled RAG copies. AI-assisted tools (attack generation, attacker–judge loops) — verify judges, keep humans for novelty.
Findings and mitigations
Triage: severity · exploitability · reproducibility (success rate) · business impact. Low rate + cheap attempts can still be critical. Vulnerability = adversary gains something / control bypassed. Limitation = ordinary error (hallucination, over-refusal). Map to OWASP + ATLAS + internal control. Evidence: versions, inputs, outputs, success rate, steps. Communicate: engineers (repro, fix) · owners (likelihood, impact, options) · board (consequence, decision). Mitigate — prefer architectural fixes; prioritise impact × exploitability; validate with regression; add to suite; monitor residual risk with KRIs.
Exam-day reflexes
- "Instructions hidden in an email the assistant reads" → indirect injection.
- "Was this patient in the training set?" → membership inference → DP.
- "Model output into shell/browser" → LLM05.
- "Pin versions before testing" → reproducibility.
- "Invented clause, no attacker" → limitation, not vulnerability.