AI Threat Modelling, 25 min read

4.R Review and references

In this lesson you will:

  • Recap the main points of Module 4
  • Test yourself on six review questions
  • Find the sources cited in this module

This page closes Module 4, AI Threat Modelling. Citation numbers in square brackets in the sections refer to the reference list at the end of this page.

Module summary

  • AI systems differ from traditional systems in seven ways that matter for threat modelling: probabilistic behaviour; instructions and data in one channel; data acting as code; new assets (weights, datasets, prompts, embeddings, memory); a wider supply chain; software that acts with delegated authority; and new failure modes that need evaluation and monitoring.
  • Treat the model as an untrusted component. Enforce security outside it; prompts and guardrail models are layers, not controls.
  • Model the whole lifecycle (data, training, evaluation, packaging, deployment, operation), mark every untrusted-content entry point, and list every tool with its authority.
  • STRIDE still works. Learn the AI forms: deepfakes and agent impersonation (S); poisoning, prompt injection and memory poisoning (T); untraceable agent actions (R); extraction, leakage and oversharing (I); unbounded consumption and loops (D); excessive agency and confused deputies (E).
  • Use the OWASP Top 10 for LLM Applications (2026), the OWASP Top 10 for Agentic Applications (ASI01–ASI10), MITRE ATLAS, NIST AI 100-2 and MAESTRO for what each does best, and always state the edition of an OWASP ID.
  • ML model example: most risk sits in data, labelling, feedback loops, base models and front-door evasion. Review before learning; sign and scan models; hide scores; limit attempts.
  • RAG example: most risk sits in what reaches the context and what happens to the output. Curate the corpus; filter retrieval by session identity; bind tools to the session; render output safely; ground answers.
  • Agent example: most risk comes from untrusted content combined with permissions. Apply least agency, delegated identity, approval gates with original evidence, isolation of untrusted content, controlled memory, secured MCP and A2A peers, circuit breakers and full traces.

Review questions

1. What is the root cause that makes indirect prompt injection possible?

  • A. Hosted LLMs do not encrypt prompts in transit
  • B. The model processes instructions and untrusted data in the same context without a reliable way to separate them
  • C. Vector stores do not support access control
  • D. System prompts are visible to users

2. A fraud model's training set is updated monthly with cases labelled by front-line reviewers, with no further check. Which threat does this MOST directly create?

  • A. Model extraction
  • B. Poisoning through the feedback loop
  • C. Membership inference
  • D. Unbounded consumption

3. In a RAG assistant, which control BEST prevents one customer seeing another customer's documents?

  • A. A system prompt telling the model to respect confidentiality
  • B. Filtering retrieval by the authenticated user's permissions before chunks reach the model
  • C. An output classifier that flags personal names
  • D. Encrypting the vector store at rest

4. Which OWASP Agentic Top 10 entry describes an agent whose long-term memory has been seeded with false facts by an attacker?

  • A. ASI01 Agent Goal Hijack
  • B. ASI04 Agentic Supply Chain Vulnerabilities
  • C. ASI06 Memory and Context Poisoning
  • D. ASI10 Rogue Agents

5. An agent acts with a service account that can refund any transaction, while each human analyst is limited to ₹25,000. What is the BEST fix?

  • A. Add a system-prompt rule limiting refunds to ₹25,000
  • B. Have the agent act with a delegated, case-scoped token of the assigning analyst, with tool-level caps
  • C. Log every refund the agent makes
  • D. Use a larger, more capable model

6. What does MAESTRO add to STRIDE for agentic systems?

  • A. A numerical risk score for each threat
  • B. A seven-layer decomposition that highlights threats within and across layers of an agentic system
  • C. A list of CVEs in agent frameworks
  • D. A replacement for data flow diagrams
Show answers
  1. B. Instructions and data share one channel; untrusted text in the context can act as instructions.
  2. B. Anyone who reaches human review can influence labels that become training data.
  3. B. If the model never receives unauthorised chunks, it cannot reveal them. A and C are layers; D protects against a different threat.
  4. C. Memory and context poisoning.
  5. B. Delegated, scoped identity removes the confused deputy; tool caps bound the damage. A is guidance, C is detection only.
  6. B. MAESTRO structures the system into layers (foundation models to ecosystem) and prompts cross-layer analysis; it works alongside STRIDE.

References

  1. OWASP GenAI Security Project. OWASP Top 10 for LLM Applications 2026. August 2026. https://genai.owasp.org/
  2. OWASP GenAI Security Project. OWASP Top 10 for LLM Applications 2025. November 2024.
  3. OWASP GenAI Security Project, Agentic Security Initiative. OWASP Top 10 for Agentic Applications 2026. December 2025.
  4. MITRE. ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems. https://atlas.mitre.org/
  5. NIST. AI 100-2 E2025: Adversarial Machine Learning — A Taxonomy and Terminology of Attacks and Mitigations. March 2025.
  6. Huang, K. "Agentic AI Threat Modeling Framework: MAESTRO". Cloud Security Alliance, February 2025.
  7. Model Context Protocol. Specification. https://modelcontextprotocol.io/
  8. Agent2Agent (A2A) Protocol. Specification. https://a2a-protocol.org/
  9. Greshake, K. et al. "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection". AISec '23, 2023.
  10. Willison, S. "The Dual LLM pattern for building AI assistants that can resist prompt injection". April 2023.
  11. NIST. AI 600-1: Artificial Intelligence Risk Management Framework: Generative AI Profile. July 2024.