AI Threat Modelling, Red Teaming & Assurance, 10 min read

Ready reckoner

Threat modelling

STRIDE AI example
Spoofing Fake agent / MCP server, voice deepfake
Tampering Poisoned data, edited weights or prompts
Repudiation Unattributable agent actions
Info disclosure Injection exfil, data extraction, prompt leakage
DoS Token exhaustion, agent loops
EoP Injection drives tools beyond user's rights
  • PASTA: 7 stages, risk-centric, attack simulation — for critical systems.
  • MAESTRO (CSA): 7 layers — foundation models · data ops · agent frameworks · deployment · evaluation & observability · security & compliance · agent ecosystem. Watch cross-layer threats.
  • Actors: criminals · nation states · competitors · hacktivists · abusers · malicious & negligent insiders · non-human insiders (manipulated agents).
  • Intel: MITRE ATLAS, OWASP, incident databases, provider advisories.

Adversarial ML

Attack When Goal
Evasion Inference Wrong decision on crafted input
Poisoning / clean-label Training Degrade or target
Backdoor Training / distribution Trigger behaviour
Extraction Inference Copy the model
Inversion Inference Reconstruct training data
Membership inference Inference Was X in training?
Data extraction Inference Regurgitate PII/secrets
Availability Inference Sponge inputs, token exhaustion, denial of wallet

White/grey/black-box; transferability via surrogates. Defences: adversarial training, DP, output minimisation, rate limits, caps, multiple signals.

GenAI threats

  • Direct injection (user) · indirect injection (content the system reads) · jailbreak (bypass safety = content). Injection = control.
  • OWASP LLM Top 10 (2025): 01 injection · 02 sensitive disclosure · 03 supply chain · 04 data & model poisoning · 05 improper output handling · 06 excessive agency · 07 system prompt leakage · 08 vector & embedding weaknesses · 09 misinformation · 10 unbounded consumption.
  • RAG: poisoning, ranking manipulation, ACL bypass, cross-tenant leakage, embedding inversion.
  • Agents: tool misuse, memory poisoning, goal manipulation, confused deputy, spoofing, cascades, oversight flooding, covert exfil (markdown image URLs).
  • Multimodal: text in images, hidden audio, white-on-white PDF text, deepfakes.

Red teaming

Scope: system + pinned versions · objectives · actors · in/out of scope · rules of engagement · reporting · retest. Loop: recon → baseline → manual → automated → chain → report → retest. Report success rates (e.g. 14/50). Test whole system, every entry point; canaries; controlled RAG copies. AI-assisted tools (attack generation, attacker–judge loops) — verify judges, keep humans for novelty.

Findings and mitigations

Triage: severity · exploitability · reproducibility (success rate) · business impact. Low rate + cheap attempts can still be critical. Vulnerability = adversary gains something / control bypassed. Limitation = ordinary error (hallucination, over-refusal). Map to OWASP + ATLAS + internal control. Evidence: versions, inputs, outputs, success rate, steps. Communicate: engineers (repro, fix) · owners (likelihood, impact, options) · board (consequence, decision). Mitigate — prefer architectural fixes; prioritise impact × exploitability; validate with regression; add to suite; monitor residual risk with KRIs.

Exam-day reflexes

  • "Instructions hidden in an email the assistant reads" → indirect injection.
  • "Was this patient in the training set?" → membership inference → DP.
  • "Model output into shell/browser" → LLM05.
  • "Pin versions before testing" → reproducibility.
  • "Invented clause, no attacker" → limitation, not vulnerability.

Visual summary

Injection types
Remember Injection = control; jailbreak = content.
MAESTRO layers
Remember Seven layers; cross-layer threats matter most.