Threats to GenAI, RAG, agentic and multimodal systems
In this lesson you will:
- Distinguish direct and indirect prompt injection, and jailbreaks
- Map GenAI risks to the OWASP Top 10 for LLM Applications
- Describe threats specific to RAG, agentic and multimodal systems
- Use MITRE ATLAS tactics to describe AI attack chains
Generative systems add a new attack surface: natural-language instructions. Anyone who can get text in front of the model may be able to steer it.
1. Prompt injection and jailbreaks
- Direct prompt injection — the user types instructions that override the system's intended behaviour ("Ignore previous instructions and…").
- Indirect prompt injection — instructions are planted in content the system processes: web pages, emails, documents, tool results, images, even file names. The user may be the victim, not the attacker.
- Jailbreaks — techniques that bypass a model's safety training to produce prohibited content: role-play, hypothetical framing, encoding (base64, other languages), many-shot examples, multi-turn escalation, adversarial suffixes found by optimisation.
Injection is about control (making the system do what the attacker wants); jailbreaking is about content (making the model say what it shouldn't). They often combine.
2. OWASP Top 10 for LLM Applications (2025)
| ID | Risk | One-line summary |
|---|---|---|
| LLM01 | Prompt injection | Inputs alter the model's behaviour in unintended ways |
| LLM02 | Sensitive information disclosure | Model reveals personal data, secrets or confidential information |
| LLM03 | Supply chain | Compromised models, datasets, adapters or dependencies |
| LLM04 | Data and model poisoning | Manipulated training, fine-tuning or embedding data |
| LLM05 | Improper output handling | Unvalidated output passed to downstream systems |
| LLM06 | Excessive agency | Too much functionality, permission or autonomy |
| LLM07 | System prompt leakage | Exposure of system prompts that contain sensitive logic or secrets |
| LLM08 | Vector and embedding weaknesses | Attacks on RAG stores: poisoning, leakage, inversion, access-control gaps |
| LLM09 | Misinformation | Hallucinated or misleading output that people rely on |
| LLM10 | Unbounded consumption | Excessive resource use: denial of service, denial of wallet, extraction |
OWASP also publishes guidance on agentic AI threats, reflecting risks such as memory poisoning, tool misuse, identity spoofing and cascading failures in multi-agent systems.
3. RAG-specific threats
- Knowledge-base poisoning — plant documents crafted to be retrieved for certain queries and to carry false facts or injected instructions.
- Retrieval manipulation — content optimised to rank highly (SEO for embeddings).
- Access-control bypass — the assistant retrieves documents the user isn't entitled to see.
- Cross-tenant leakage — shared indexes without tenant isolation.
- Embedding inversion — reconstructing text from stored vectors.
- Citation spoofing — answers that cite real sources for claims those sources don't make.
4. Agentic threats
- Tool misuse via injection — an agent reading a malicious web page sends data to the attacker's server.
- Memory poisoning — persistent instructions planted in long-term memory.
- Goal manipulation — gradually steering the agent's objective over a long task.
- Privilege compromise / confused deputy — the agent uses its broad rights for a less-privileged requester.
- Identity spoofing — impersonated agents or MCP servers.
- Cascading failures and hallucination propagation — one agent's error or compromise spreads through a multi-agent chain.
- Overwhelming human oversight — flooding approvers with requests until they rubber-stamp.
- Data exfiltration channels — rendering a markdown image whose URL encodes stolen data; calling a "harmless" webhook tool.
5. Multimodal threats
- Visual prompt injection — instructions embedded in images (visible text, near-invisible text, adversarial patterns).
- Audio injection — hidden commands in audio, including ultrasonic or masked speech.
- Document injection — white-on-white text, metadata, hidden layers in PDFs and office files.
- Deepfakes — synthetic voice or video used to deceive people or AI-based verification (e.g. voice-cloned calls to a helpdesk).
6. Describing attacks with MITRE ATLAS
MITRE ATLAS is a knowledge base of adversary tactics and techniques against AI systems, modelled on ATT&CK. Its tactics include reconnaissance, resource development, initial access, AI model access, execution, persistence, privilege escalation, defence evasion, credential access, discovery, collection, AI attack staging, exfiltration and impact. Mapping a red-team finding or incident to ATLAS techniques gives a common language with other teams and threat intelligence.
Worked example — an indirect injection kill chain against a browsing agent
| Step | What happens | ATLAS-style tactic |
|---|---|---|
| 1 | Attacker publishes a product-review page containing hidden text addressed to AI assistants | Resource development |
| 2 | A user asks their browsing agent to compare products; the agent reads the page | Initial access (indirect injection) |
| 3 | Hidden text instructs the agent to fetch the user's saved addresses from a connected account | Execution / privilege abuse |
| 4 | Agent renders a markdown image whose URL contains the addresses, sending them to the attacker's server | Exfiltration |
| 5 | Attacker sells the data | Impact |
Breaks in the chain: treat page content as untrusted data (step 2); the browsing task's scope excludes the account connector (step 3); block rendering of external images or URL-encoded data in agent output (step 4); log and alert on unusual outbound URLs (step 4).
🔐 Security lens
For every GenAI feature, ask: "Whose text can reach the model, and what could the model do if that text were hostile?" If the answer to the second part is "not much", injection is a nuisance. If it's "send data or take actions", it's critical.
Key takeaways
- Direct injection comes from users; indirect injection hides in processed content; jailbreaks bypass safety training.
- The OWASP LLM Top 10 (2025): injection, sensitive disclosure, supply chain, poisoning, improper output handling, excessive agency, system prompt leakage, vector/embedding weaknesses, misinformation, unbounded consumption.
- RAG threats: poisoning, retrieval manipulation, ACL bypass, cross-tenant leakage, embedding inversion.
- Agentic threats: tool misuse, memory poisoning, goal manipulation, confused deputy, spoofing, cascades, oversight flooding, covert exfiltration.
- Multimodal inputs carry hidden instructions; ATLAS describes attack chains in shared language.
Self-check
- A user's email assistant follows instructions hidden in an incoming email. Direct or indirect injection?
- Which OWASP LLM risk covers model output passed unsanitised to a shell?
- Give two ways an agent could exfiltrate data without an obvious "send" tool.