Triage, reporting and validating mitigations
In this lesson you will:
- Triage findings by severity, exploitability, reproducibility and business impact
- Distinguish security vulnerabilities from model limitations
- Correlate findings with MITRE ATLAS and the OWASP LLM Top 10
- Document evidence and reproduction steps for non-deterministic systems
- Communicate risk to technical and non-technical stakeholders
- Prioritise, implement and validate mitigations and monitor residual risk
A red team's value is realised only when findings are understood, prioritised, fixed and verified. AI findings bring two special challenges: non-deterministic reproduction, and telling a vulnerability apart from a model simply being imperfect.
1. Triage
| Dimension | Questions |
|---|---|
| Severity / impact | What is the worst realistic consequence — data exposure, financial loss, safety harm, regulatory breach? |
| Exploitability | Who can do it (anonymous user, authenticated user, insider)? How much skill and effort? Can it be automated? |
| Reproducibility | What is the success rate over N attempts? Does it depend on specific versions or settings? |
| Business impact | Which processes, customers and obligations are affected? How many users are exposed? |
A 5% success rate is serious if attempts are free and automated: 100 tries yields about 5 successes.
2. Vulnerability or limitation?
| Finding | Classification | Owner and response |
|---|---|---|
| Assistant reveals another customer's data when prompted | Security vulnerability (access-control failure, injection) | Security + engineering; urgent fix |
| Agent can be made to issue unapproved refunds | Security vulnerability (excessive agency) | Security + engineering; urgent fix |
| Assistant occasionally gives a wrong policy date | Model limitation (hallucination) | Product/model owner; grounding, evaluation, user guidance |
| Model refuses some legitimate medical questions | Model limitation (over-refusal) | Product owner; tuning |
| Hallucinated package name that attackers could register | Limitation that creates a security exposure | Both: dependency controls plus grounding |
The test: does an adversary gain something they shouldn't (access, action, data, cost to you), or does the system bypass a control? If yes, treat it as a vulnerability. If it's the model's ordinary error rate with no adversary advantage, it's a limitation — still important, but owned and fixed differently.
3. Mapping to frameworks
Tag each finding with:
- OWASP LLM Top 10 category (e.g. LLM01 prompt injection, LLM06 excessive agency).
- MITRE ATLAS tactic and technique.
- Internal control that failed or was missing.
- Regulatory requirement affected, where relevant (e.g. EU AI Act robustness and cybersecurity).
Mapping helps spot patterns across systems ("six findings this quarter are excessive agency"), compare with threat intelligence, and report consistently.
4. Evidence and reproducibility
For non-deterministic systems, a good finding record includes:
- System versions: model, prompt, guardrails, retrieval index, tools, configuration (temperature, etc.).
- Exact inputs, including multi-turn history, uploaded files and planted documents.
- Outputs and tool calls, with timestamps.
- Success rate across a stated number of attempts.
- Preconditions (account type, data present).
- Clear reproduction steps, and a minimal reproducing case if possible.
- Handling notes: findings may contain harmful content or exploit details — restrict distribution.
5. Communicating risk
- Engineers — technical detail, reproduction steps, root cause, suggested fixes.
- Risk and system owners — scenario, likelihood, impact, options and residual risk.
- Executives and boards — business consequence in plain language, trend, decision needed.
Avoid hype ("the AI was hacked") and avoid minimising ("it only works 5% of the time"). State what an attacker could achieve, how easily, and what it would cost.
6. Mitigations
| Finding type | Targeted mitigations |
|---|---|
| Evasion / adversarial inputs | Adversarial training, input preprocessing, ensembles, additional independent signals |
| Poisoning / backdoors | Provenance tracking, data validation, versioning, trigger testing |
| Prompt injection | Context isolation, untrusted-content marking, least-privilege tools, approval gates |
| Sensitive disclosure | Permission-aware retrieval, output DLP, minimisation |
| Extraction / inversion | Rate limiting, output minimisation, watermarking, differential privacy |
| Unsafe output handling | Escaping, parameterisation, sandboxing |
| Availability abuse | Rate limits, caps, budgets, timeouts |
| Synthetic content misuse | Output watermarking and provenance |
Prioritise by risk (impact × exploitability), feasibility and business context. Quick wins (scope reductions, rate limits) can come first while deeper fixes (re-architecture, retraining) are planned.
Validate with regression testing: rerun the original attacks and their variants against the fixed system and confirm the success rate falls to an acceptable level — then add them to the automated test suite permanently.
Monitor residual risk: some risk remains (injection can't be fully eliminated). Track it with KRIs — blocked attempt rates, guardrail hits, anomaly alerts — and record risk acceptance where appropriate.
Worked example — writing up a finding
| Field | Content |
|---|---|
| Title | Indirect injection via uploaded claim documents leads to payment-detail change |
| Classification | Security vulnerability · OWASP LLM01 + LLM06 · ATLAS: initial access via prompt injection → impact |
| Versions | Agent 2.3, prompt v14, guardrails v6, staging |
| Success rate | 9 of 50 attempts (18%) with the documented payload; 0 of 50 with plain documents |
| Impact | Payout to attacker-controlled account; up to ₹2 lakh per claim before approval threshold |
| Exploitability | Any claimant; no special skills; automatable |
| Root cause | Agent can call update_payee without approval; document text not marked as untrusted |
| Fix | Remove update_payee from the agent (payee changes via verified customer flow only); mark document content as data; add approval for any payment > ₹0 to a new payee |
| Validation | Retest: 0 of 200 attempts; payload variants added to regression suite |
| Residual risk | Model still sometimes proposes the change (3%) — blocked by policy; KRI tracked |
🔐 Security lens
Prefer fixes that make an attack pointless over fixes that make it harder. Removing a dangerous tool beats detecting a cleverer prompt.
Key takeaways
- Triage on severity, exploitability, reproducibility (success rate) and business impact.
- A vulnerability gives an adversary an advantage or bypasses a control; a limitation is ordinary error — different owners, different fixes.
- Map findings to OWASP LLM Top 10, MITRE ATLAS and internal controls.
- Record versions, inputs, outputs, success rates and reproduction steps; handle sensitive findings carefully.
- Tailor communication to engineers, owners and executives.
- Prioritise targeted, preferably architectural, mitigations; validate with regression tests; monitor residual risk.
Self-check
- Why might a finding with a 5% success rate still be rated critical?
- Classify: "the assistant sometimes invents a policy clause". Vulnerability or limitation?
- What must happen to a red-team payload after its fix is validated?