AI Threat Modelling, 10 min read

Ready reckoner

Traditional vs AI: seven differences

Aspect Traditional AI So…
Behaviour Deterministic Probabilistic Evaluate many inputs; assume errors
Instructions vs data Separated One context Any untrusted text = possible instruction
Code Code Code + data + weights Datasets and pipelines are attack surface
Assets Code, data, secrets + weights, datasets, prompts, embeddings, memory Protect and inventory them
Supply chain Libraries, images + base models, datasets, adapters, MCP servers, peer agents Pin, scan, sign, allow-list
Who acts Users, services + agents with delegated authority Least agency; confused deputy
Validation Tests pass/fail Evals, red teaming, monitoring Re-test on model/prompt/tool change

Golden rule: treat the model as untrusted; enforce security outside it. Prompts and guardrail models are layers.

Lifecycle threats

Data (poisoning, PII) → Train (backdoors) → Evaluate (weak evals) → Package (model swap, unsafe pickle) → Deploy (injection, evasion, theft) → Operate (drift, feedback poisoning).

STRIDE for AI

S deepfakes, agent/tool impersonation · T poisoning, prompt injection, memory poisoning · R untraceable agent actions · I extraction, leakage, RAG oversharing, hidden context · D unbounded consumption, loops · E excessive agency, tool misuse, confused deputy.

OWASP LLM Top 10 (2026)

01 Prompt Injection · 02 Sensitive Info Disclosure · 03 Excessive Agency · 04 Supply Chain · 05 Data & Model Poisoning · 06 Unbounded Consumption · 07 Misinformation · 08 Hidden Context Exposure · 09 Vector & Embedding Weaknesses · 10 Improper Output Handling. (2025 numbering differs — cite the edition.)

OWASP Agentic Top 10 (ASI, Dec 2025)

01 Goal Hijack · 02 Tool Misuse · 03 Identity & Privilege Abuse · 04 Agentic Supply Chain · 05 Unexpected Code Execution · 06 Memory & Context Poisoning · 07 Insecure Inter-Agent Comms · 08 Cascading Failures · 09 Human-Agent Trust Exploitation · 10 Rogue Agents. Principles: least agency, strong observability.

Other frameworks

MITRE ATLAS — adversary tactics/techniques + case studies (attack paths, red teams). NIST AI 100-2 E2025 — evasion, poisoning, privacy (incl. extraction, membership inference), GenAI injection/misuse. MAESTRO (CSA) — 7 layers: foundation models · data ops · agent frameworks · deployment/infra · evaluation/observability · security & compliance · ecosystem; look for cross-layer threats.

Three examples

ML model RAG assistant Agent
Risk enters via Data, labels, feedback, base model, evasion Retrieved content, prompts, output Any text read + permissions
Key controls Review before learning; pinned/scanned/signed models; liveness; hide scores; attempt limits Curated corpus; permission-aware retrieval; session-bound tools; safe rendering; grounding Tool/permission table; parameter-free tools; caps; delegated identity; reader/actor split; approvals with original evidence; MCP/A2A auth; circuit breakers; traces

Agent question: if an attacker controlled any text this agent reads, what is the worst it could do with its permissions?