AI Threat Modelling, Red Teaming & Assurance, 30 min read

Triage, reporting and validating mitigations

In this lesson you will:

  • Triage findings by severity, exploitability, reproducibility and business impact
  • Distinguish security vulnerabilities from model limitations
  • Correlate findings with MITRE ATLAS and the OWASP LLM Top 10
  • Document evidence and reproduction steps for non-deterministic systems
  • Communicate risk to technical and non-technical stakeholders
  • Prioritise, implement and validate mitigations and monitor residual risk

A red team's value is realised only when findings are understood, prioritised, fixed and verified. AI findings bring two special challenges: non-deterministic reproduction, and telling a vulnerability apart from a model simply being imperfect.

1. Triage

Triage of AI findings by severity, exploitability, reproducibility and business impact.
Figure 7.9 Four triage dimensions. A low success rate does not mean low severity if the impact is severe and attempts are cheap.
Dimension Questions
Severity / impact What is the worst realistic consequence — data exposure, financial loss, safety harm, regulatory breach?
Exploitability Who can do it (anonymous user, authenticated user, insider)? How much skill and effort? Can it be automated?
Reproducibility What is the success rate over N attempts? Does it depend on specific versions or settings?
Business impact Which processes, customers and obligations are affected? How many users are exposed?

A 5% success rate is serious if attempts are free and automated: 100 tries yields about 5 successes.

2. Vulnerability or limitation?

Finding Classification Owner and response
Assistant reveals another customer's data when prompted Security vulnerability (access-control failure, injection) Security + engineering; urgent fix
Agent can be made to issue unapproved refunds Security vulnerability (excessive agency) Security + engineering; urgent fix
Assistant occasionally gives a wrong policy date Model limitation (hallucination) Product/model owner; grounding, evaluation, user guidance
Model refuses some legitimate medical questions Model limitation (over-refusal) Product owner; tuning
Hallucinated package name that attackers could register Limitation that creates a security exposure Both: dependency controls plus grounding

The test: does an adversary gain something they shouldn't (access, action, data, cost to you), or does the system bypass a control? If yes, treat it as a vulnerability. If it's the model's ordinary error rate with no adversary advantage, it's a limitation — still important, but owned and fixed differently.

3. Mapping to frameworks

Tag each finding with:

  • OWASP LLM Top 10 category (e.g. LLM01 prompt injection, LLM06 excessive agency).
  • MITRE ATLAS tactic and technique.
  • Internal control that failed or was missing.
  • Regulatory requirement affected, where relevant (e.g. EU AI Act robustness and cybersecurity).

Mapping helps spot patterns across systems ("six findings this quarter are excessive agency"), compare with threat intelligence, and report consistently.

4. Evidence and reproducibility

For non-deterministic systems, a good finding record includes:

  • System versions: model, prompt, guardrails, retrieval index, tools, configuration (temperature, etc.).
  • Exact inputs, including multi-turn history, uploaded files and planted documents.
  • Outputs and tool calls, with timestamps.
  • Success rate across a stated number of attempts.
  • Preconditions (account type, data present).
  • Clear reproduction steps, and a minimal reproducing case if possible.
  • Handling notes: findings may contain harmful content or exploit details — restrict distribution.

5. Communicating risk

The same finding communicated to engineers, risk owners and executives.
Figure 7.10 One finding, three audiences. Same facts, different emphasis.
  • Engineers — technical detail, reproduction steps, root cause, suggested fixes.
  • Risk and system owners — scenario, likelihood, impact, options and residual risk.
  • Executives and boards — business consequence in plain language, trend, decision needed.

Avoid hype ("the AI was hacked") and avoid minimising ("it only works 5% of the time"). State what an attacker could achieve, how easily, and what it would cost.

6. Mitigations

Mitigations mapped to finding types.
Figure 7.11 Mitigations matched to finding types. Architectural fixes usually beat filters.
Finding type Targeted mitigations
Evasion / adversarial inputs Adversarial training, input preprocessing, ensembles, additional independent signals
Poisoning / backdoors Provenance tracking, data validation, versioning, trigger testing
Prompt injection Context isolation, untrusted-content marking, least-privilege tools, approval gates
Sensitive disclosure Permission-aware retrieval, output DLP, minimisation
Extraction / inversion Rate limiting, output minimisation, watermarking, differential privacy
Unsafe output handling Escaping, parameterisation, sandboxing
Availability abuse Rate limits, caps, budgets, timeouts
Synthetic content misuse Output watermarking and provenance

Prioritise by risk (impact × exploitability), feasibility and business context. Quick wins (scope reductions, rate limits) can come first while deeper fixes (re-architecture, retraining) are planned.

Validate with regression testing: rerun the original attacks and their variants against the fixed system and confirm the success rate falls to an acceptable level — then add them to the automated test suite permanently.

Monitor residual risk: some risk remains (injection can't be fully eliminated). Track it with KRIs — blocked attempt rates, guardrail hits, anomaly alerts — and record risk acceptance where appropriate.

Worked example — writing up a finding

Field Content
Title Indirect injection via uploaded claim documents leads to payment-detail change
Classification Security vulnerability · OWASP LLM01 + LLM06 · ATLAS: initial access via prompt injection → impact
Versions Agent 2.3, prompt v14, guardrails v6, staging
Success rate 9 of 50 attempts (18%) with the documented payload; 0 of 50 with plain documents
Impact Payout to attacker-controlled account; up to ₹2 lakh per claim before approval threshold
Exploitability Any claimant; no special skills; automatable
Root cause Agent can call update_payee without approval; document text not marked as untrusted
Fix Remove update_payee from the agent (payee changes via verified customer flow only); mark document content as data; add approval for any payment > ₹0 to a new payee
Validation Retest: 0 of 200 attempts; payload variants added to regression suite
Residual risk Model still sometimes proposes the change (3%) — blocked by policy; KRI tracked

🔐 Security lens

Prefer fixes that make an attack pointless over fixes that make it harder. Removing a dangerous tool beats detecting a cleverer prompt.

Key takeaways

  • Triage on severity, exploitability, reproducibility (success rate) and business impact.
  • A vulnerability gives an adversary an advantage or bypasses a control; a limitation is ordinary error — different owners, different fixes.
  • Map findings to OWASP LLM Top 10, MITRE ATLAS and internal controls.
  • Record versions, inputs, outputs, success rates and reproduction steps; handle sensitive findings carefully.
  • Tailor communication to engineers, owners and executives.
  • Prioritise targeted, preferably architectural, mitigations; validate with regression tests; monitor residual risk.

Self-check

  1. Why might a finding with a 5% success rate still be rated critical?
  2. Classify: "the assistant sometimes invents a policy clause". Vulnerability or limitation?
  3. What must happen to a red-team payload after its fix is validated?