AI Threat Modelling, Red Teaming & Assurance, 30 min read

Threats to GenAI, RAG, agentic and multimodal systems

In this lesson you will:

  • Distinguish direct and indirect prompt injection, and jailbreaks
  • Map GenAI risks to the OWASP Top 10 for LLM Applications
  • Describe threats specific to RAG, agentic and multimodal systems
  • Use MITRE ATLAS tactics to describe AI attack chains

Generative systems add a new attack surface: natural-language instructions. Anyone who can get text in front of the model may be able to steer it.

1. Prompt injection and jailbreaks

Direct injection, indirect injection and jailbreak compared.
Figure 7.5 Direct injection comes from the user; indirect injection hides in content the system retrieves; jailbreaks target the model's safety training.
  • Direct prompt injection — the user types instructions that override the system's intended behaviour ("Ignore previous instructions and…").
  • Indirect prompt injection — instructions are planted in content the system processes: web pages, emails, documents, tool results, images, even file names. The user may be the victim, not the attacker.
  • Jailbreaks — techniques that bypass a model's safety training to produce prohibited content: role-play, hypothetical framing, encoding (base64, other languages), many-shot examples, multi-turn escalation, adversarial suffixes found by optimisation.

Injection is about control (making the system do what the attacker wants); jailbreaking is about content (making the model say what it shouldn't). They often combine.

2. OWASP Top 10 for LLM Applications (2025)

ID Risk One-line summary
LLM01 Prompt injection Inputs alter the model's behaviour in unintended ways
LLM02 Sensitive information disclosure Model reveals personal data, secrets or confidential information
LLM03 Supply chain Compromised models, datasets, adapters or dependencies
LLM04 Data and model poisoning Manipulated training, fine-tuning or embedding data
LLM05 Improper output handling Unvalidated output passed to downstream systems
LLM06 Excessive agency Too much functionality, permission or autonomy
LLM07 System prompt leakage Exposure of system prompts that contain sensitive logic or secrets
LLM08 Vector and embedding weaknesses Attacks on RAG stores: poisoning, leakage, inversion, access-control gaps
LLM09 Misinformation Hallucinated or misleading output that people rely on
LLM10 Unbounded consumption Excessive resource use: denial of service, denial of wallet, extraction

OWASP also publishes guidance on agentic AI threats, reflecting risks such as memory poisoning, tool misuse, identity spoofing and cascading failures in multi-agent systems.

3. RAG-specific threats

Attack paths against a RAG system.
Figure 7.6 RAG attack paths: poison what gets retrieved, abuse what retrieval can reach, or attack the embeddings themselves.
  • Knowledge-base poisoning — plant documents crafted to be retrieved for certain queries and to carry false facts or injected instructions.
  • Retrieval manipulation — content optimised to rank highly (SEO for embeddings).
  • Access-control bypass — the assistant retrieves documents the user isn't entitled to see.
  • Cross-tenant leakage — shared indexes without tenant isolation.
  • Embedding inversion — reconstructing text from stored vectors.
  • Citation spoofing — answers that cite real sources for claims those sources don't make.

4. Agentic threats

  • Tool misuse via injection — an agent reading a malicious web page sends data to the attacker's server.
  • Memory poisoning — persistent instructions planted in long-term memory.
  • Goal manipulation — gradually steering the agent's objective over a long task.
  • Privilege compromise / confused deputy — the agent uses its broad rights for a less-privileged requester.
  • Identity spoofing — impersonated agents or MCP servers.
  • Cascading failures and hallucination propagation — one agent's error or compromise spreads through a multi-agent chain.
  • Overwhelming human oversight — flooding approvers with requests until they rubber-stamp.
  • Data exfiltration channels — rendering a markdown image whose URL encodes stolen data; calling a "harmless" webhook tool.

5. Multimodal threats

  • Visual prompt injection — instructions embedded in images (visible text, near-invisible text, adversarial patterns).
  • Audio injection — hidden commands in audio, including ultrasonic or masked speech.
  • Document injection — white-on-white text, metadata, hidden layers in PDFs and office files.
  • Deepfakes — synthetic voice or video used to deceive people or AI-based verification (e.g. voice-cloned calls to a helpdesk).

6. Describing attacks with MITRE ATLAS

MITRE ATLAS is a knowledge base of adversary tactics and techniques against AI systems, modelled on ATT&CK. Its tactics include reconnaissance, resource development, initial access, AI model access, execution, persistence, privilege escalation, defence evasion, credential access, discovery, collection, AI attack staging, exfiltration and impact. Mapping a red-team finding or incident to ATLAS techniques gives a common language with other teams and threat intelligence.

Worked example — an indirect injection kill chain against a browsing agent

Step What happens ATLAS-style tactic
1 Attacker publishes a product-review page containing hidden text addressed to AI assistants Resource development
2 A user asks their browsing agent to compare products; the agent reads the page Initial access (indirect injection)
3 Hidden text instructs the agent to fetch the user's saved addresses from a connected account Execution / privilege abuse
4 Agent renders a markdown image whose URL contains the addresses, sending them to the attacker's server Exfiltration
5 Attacker sells the data Impact

Breaks in the chain: treat page content as untrusted data (step 2); the browsing task's scope excludes the account connector (step 3); block rendering of external images or URL-encoded data in agent output (step 4); log and alert on unusual outbound URLs (step 4).

🔐 Security lens

For every GenAI feature, ask: "Whose text can reach the model, and what could the model do if that text were hostile?" If the answer to the second part is "not much", injection is a nuisance. If it's "send data or take actions", it's critical.

Key takeaways

  • Direct injection comes from users; indirect injection hides in processed content; jailbreaks bypass safety training.
  • The OWASP LLM Top 10 (2025): injection, sensitive disclosure, supply chain, poisoning, improper output handling, excessive agency, system prompt leakage, vector/embedding weaknesses, misinformation, unbounded consumption.
  • RAG threats: poisoning, retrieval manipulation, ACL bypass, cross-tenant leakage, embedding inversion.
  • Agentic threats: tool misuse, memory poisoning, goal manipulation, confused deputy, spoofing, cascades, oversight flooding, covert exfiltration.
  • Multimodal inputs carry hidden instructions; ATLAS describes attack chains in shared language.

Self-check

  1. A user's email assistant follows instructions hidden in an incoming email. Direct or indirect injection?
  2. Which OWASP LLM risk covers model output passed unsanitised to a shell?
  3. Give two ways an agent could exfiltrate data without an obvious "send" tool.