6.1 AI threat modeling and intelligence
In this lesson you will:
- Pick a threat modeling method that fits the system, and cover the whole AI pipeline with it.
- Describe who is likely to attack an AI system, including other AI agents.
- Name the classic attacks on machine learning and the defenses for each.
- List the extra threats that come with generative, RAG, agentic and multimodal systems.
- Tell a credible robustness claim from marketing.
Opening case: Mortgage Copilot
Meridian plans Mortgage Copilot, an agent for mortgage brokers. A broker uploads a client's documents (payslips, bank statements, ID). The agent extracts the figures, retrieves Meridian's lending policy from a knowledge base, checks affordability, drafts a decision-in-principle and, if the broker approves, submits the application to the loan system through a tool. It uses an external foundation model and a document-understanding model that reads scanned images.
The architect brings a one-page diagram to the threat modeling session and says: "We did STRIDE on the web app last year. Can we reuse that?"
By the end of this section you will be able to say why not, choose the method, identify the main threats and attackers, and question the vendor's claim that its document model is "robust to adversarial inputs".
Choosing a threat modeling method
Threat modeling asks four questions [17]:
- What are we building?
- What can go wrong?
- What will we do about it?
- Did we do a good job?
Three methods appear on the exam [16][17][18]:
| Method | Approach | Best for |
|---|---|---|
| STRIDE | For each component and data flow, consider Spoofing, Tampering, Repudiation, Information disclosure, Denial of service and Elevation of privilege | Quick, systematic coverage of a well-understood architecture |
| PASTA | Seven-stage, risk-centric process from business objectives through attack simulation to risk analysis | Aligning threat modeling with business impact and risk decisions |
| MAESTRO | Cloud Security Alliance framework for agentic AI. Examines threats at each of seven layers (foundation models, data operations, agent frameworks, deployment, evaluation and observability, security and compliance, agent ecosystem) and between layers | Multi-agent and agentic systems where threats cross layers |
STRIDE still works for AI if you apply it to AI components. Figure 6.1 gives AI examples for each category.
MAESTRO adds what STRIDE on a diagram tends to miss: threats that arise between layers, such as a poisoned document in data operations steering a tool call in the agent framework. Figure 6.2 shows its layers.
Whatever the method:
- cover the whole AI pipeline: data, training, supply chain, deployment, inference, agents and tools (Figure 6.3);
- use threat libraries rather than memory: MITRE ATLAS for adversary techniques with real-world case studies [13], the OWASP Top 10 lists for LLM and agentic applications [14][15], and the NIST adversarial ML taxonomy [12].
Worked example: why last year's STRIDE is not enough
Last year's model covered the web app: login, upload, database. Mortgage Copilot adds components and flows the old diagram does not have:
| New element | Example threat | Method that surfaces it |
|---|---|---|
| Uploaded documents → document model | Tampering: a payslip image altered so the model reads 4,200 as 14,200 (evasion) | STRIDE on the new flow |
| Uploaded documents → foundation model | Elevation: hidden text in a bank statement instructs the agent to skip affordability checks (indirect injection) | STRIDE, and MAESTRO cross-layer |
| Knowledge base → agent | Tampering: an edited policy page lowers the income multiple (knowledge-base poisoning) | MAESTRO data operations → agent framework |
| Agent → loan system tool | Repudiation: submissions logged as "broker" with no record of the agent (section 4.5) | STRIDE |
| External model provider | Information disclosure: client data retained by the provider; denial of service: provider outage | STRIDE, supply-chain review |
| Business objective | Fraudulent mortgages approved at scale | PASTA stage 1 ties threats to loss |
Choice: MAESTRO as the primary method, because Mortgage Copilot is agentic and its worst threats cross layers. Use STRIDE prompts inside each layer for completeness, and borrow PASTA's first stage to rank threats by business impact.
Profiling threat actors
Profile likely attackers by motivation, capability and access [3]:
| Actor | Motivation | Capability | Typical AI attack |
|---|---|---|---|
| Opportunistic users | Curiosity, free use, mischief | Low; copy published jailbreaks | Jailbreaks, prompt-leak attempts, abuse of free tiers |
| Cybercriminals | Money | Moderate to high; tooling and services | Data theft via agents, fraud, credential harvesting from AI supply chains, "denial of wallet" |
| Competitors | Advantage | Moderate | Model extraction, scraping outputs to train rival models |
| Nation-state actors | Espionage, disruption, influence | High; patient | Supply-chain compromise, data poisoning, theft of model weights |
| Insiders | Grievance, money, carelessness | Privileged access | Weight or data theft, tampering, shadow AI leaks |
| Non-human actors | Whatever they were instructed or manipulated to do | Varies | Compromised or misaligned agents attacking other systems; automated attack tooling |
Non-human actors are new. An attacker's AI agent may probe your systems at machine speed, and your own agent can be turned against you by injected instructions. Threat models must treat agents both as targets and as threat actors.
For Mortgage Copilot, the actors that matter most are fraudsters (cybercriminals) submitting doctored documents, dishonest brokers (insiders with legitimate access and a commission motive), and the agent itself if manipulated.
The classic attack families
The NIST taxonomy organizes attacks by when they occur and what they target [12]:
| Attack | When | Target | Example | Defenses |
|---|---|---|---|---|
| Evasion | Inference | Integrity | Malware modified to pass an AI detector; an altered payslip image | Adversarial training, input transformation, ensembles, no single model for critical decisions |
| Poisoning and backdoors | Training | Integrity or availability | The postcode backdoor in section 5.6 | Provenance, data validation, robust training, backdoor detection |
| Model extraction | Inference | Confidentiality | Rebuilding a model through its API | Access control, output minimization, rate limits, detection |
| Model inversion and membership inference | Inference | Privacy | Recovering features of training data, or confirming someone was in it | Differential privacy, regularization, output minimization |
Threats specific to generative, RAG, agentic and multimodal systems
Generative AI adds its own threats, reflected in the OWASP Top 10 for LLM Applications 2025 [14]:
- prompt injection (direct and indirect);
- sensitive information disclosure;
- supply-chain risks;
- data and model poisoning;
- improper output handling (passing model output unchecked to browsers, shells or databases);
- excessive agency;
- system prompt leakage;
- vector and embedding weaknesses;
- misinformation;
- unbounded consumption.
Each architecture adds more:
- RAG: knowledge-base poisoning and retrieval oversharing. Figure 6.4 shows the attack paths.
- Agentic: tool misuse, memory poisoning, privilege compromise and cascading failures between agents [15].
- Multimodal: injections hidden in images, audio and documents. The bank-statement attack above is one.
Compute and token exhaustion
Attackers can target the economics of AI:
- very long inputs;
- prompts designed to produce maximum-length outputs;
- requests that trigger expensive tool chains;
- agents tricked into infinite loops.
Layered defenses (Module 4) include token and request limits per identity, caps on output length and agent steps, quotas and budget alerts, autoscaling maximums, and detection of abnormal consumption.
Judging robustness claims
Evasion and privacy attacks are an arms race, and claims of robustness need scrutiny. A credible robustness evaluation [27]:
- states a clear threat model: what the attacker knows and can change;
- tests against strong, adaptive attacks designed specifically to defeat the defense, not only off-the-shelf attacks;
- reports results across a range of attack strengths;
- is suspicious of defenses that seem to work by hiding gradients rather than by being truly robust.
Worked example: the document model's claim
The vendor's data sheet says the model is "robust to adversarial inputs, with 98% accuracy under attack." Questions to send back:
| Question | Why it matters |
|---|---|
| Under which threat model: pixel changes of what size? Edits to printed numbers? Re-scans? | 98% against invisible pixel noise says nothing about a fraudster editing a digit |
| Which attacks? Were any adaptive to your defense? | Standard attacks are easy to defend against selectively |
| What is accuracy at stronger attack budgets? | A single number hides a curve that may collapse |
| Can we test it with our own altered documents? | Your threat model, not theirs |
Until answered, Meridian treats the model as not robust for fraud purposes and keeps a second, independent check on income figures (for example, matching against open-banking data).
Exam lens. When a question offers "the vendor states the model is robust" as evidence, look for the answer that asks for the threat model and adaptive attack results. Robustness without a stated threat model is not evidence.
Check your understanding
Try each before reading the reasoning.
1. A team must threat model a multi-agent system where a research agent passes findings to a trading agent. Which method fits best, and why?
Reasoning: MAESTRO, because it is designed for agentic systems and examines threats between layers and agents, such as poisoned research content flowing into trading actions. STRIDE prompts can be used within it.
2. An agent is manipulated by a malicious web page into probing internal systems. In threat-actor terms, what is it?
Reasoning: a non-human actor. Your own agent, turned by injected instructions, acts as a threat actor, so threat models must treat agents as both targets and potential attackers.
3. An attacker sends thousands of prompts designed to make a model produce maximum-length outputs. What is the threat, and two defenses?
Reasoning: unbounded consumption (compute and token exhaustion, "denial of wallet"). Defenses: output length caps and per-identity token quotas, with budget alerts and abnormal-consumption detection.
Key takeaways
- Choose the method to fit the system: STRIDE for systematic coverage, PASTA for business-risk alignment, MAESTRO for agentic systems with cross-layer threats. Cover the whole AI pipeline and use threat libraries (ATLAS, OWASP, NIST).
- Profile actors by motivation, capability and access, and include non-human actors.
- Know the classic families: evasion, poisoning and backdoors, extraction, inversion and membership inference, with their defenses.
- Generative, RAG, agentic and multimodal systems add injection, oversharing, excessive agency, memory poisoning, hidden multimodal payloads and unbounded consumption.
- Credible robustness claims state a threat model, use adaptive attacks, and report results across attack strengths.