Responding to, containing and investigating AI incidents
In this lesson you will:
- Classify AI incidents with an AI-specific triage taxonomy
- Escalate across SecOps, MLOps, data science, legal and compliance
- Apply AI-specific containment, including rollback, isolation, rate limiting, pipeline suspension and agent termination
- Preserve investigative artefacts such as model states, weights, configurations and dataset snapshots
- Reconstruct attack kill chains, perform root cause analysis and consider attribution
- Eradicate threats and run safety regression testing before reintroduction
The incident response lifecycle — prepare, detect and analyse, contain, eradicate, recover, learn — applies to AI. What differs is the evidence you need, the containment levers you have, and the people you must involve.
1. AI incident triage taxonomy
| Category | Examples |
|---|---|
| Prompt injection | Direct or indirect injection causing data disclosure or unauthorised actions |
| Jailbreak / safety bypass | Prohibited or harmful content produced at scale |
| Data poisoning | Manipulated training, fine-tuning or retrieval data |
| Model theft / extraction | Weights exfiltrated; extraction campaign |
| Model tampering | Unauthorised modification of weights, prompts or configuration |
| Data disclosure | Personal or confidential data exposed via outputs, logs or retrieval |
| Agent misuse | Unauthorised actions, excessive agency, lateral movement |
| Availability / abuse | Resource exhaustion, denial of wallet |
| Supply-chain compromise | Malicious model, adapter, package or MCP server |
| Harmful outcomes | Discriminatory or unsafe decisions affecting people (may not be malicious) |
2. Escalation paths
AI incidents are multi-disciplinary:
- SecOps / incident commander — coordinates the response.
- System owner — decides on suspension, rollback and business continuity.
- MLOps / platform — executes technical containment and recovery.
- Data science — analyses model behaviour, identifies affected decisions, retrains.
- Privacy and legal — assess breach notification duties and legal exposure.
- Compliance — regulatory reporting (for example, serious-incident reporting duties for high-risk AI under the EU AI Act; in India, CERT-In directions require reporting of specified cyber incidents within six hours of noticing them, and the DPDP framework requires personal data breach notification).
- Communications — customers, partners, media.
3. AI-specific containment
| Action | When | Notes |
|---|---|---|
| Model rollback | Tampering, poisoning, bad update | To last signed, approved version |
| Disable a tool or scope | Agent misuse, injection with actions | Least disruptive way to remove impact |
| Suspend or terminate an agent | Agent compromise or runaway behaviour | Revoke its tokens too — stopping the process isn't enough |
| Endpoint isolation | Exposed or compromised serving endpoint | Remove from network or restrict to known callers |
| Rate limiting / throttling | Extraction, abuse, denial of wallet | Per identity or globally |
| Restrict retrieval | Poisoned or leaking knowledge base | Remove documents or sources from the index |
| Pipeline suspension | Poisoned data or compromised pipeline | Freeze training and promotion; prevent further contaminated models |
| Fallback mode | Primary system untrustworthy | Smaller model, rules, human queue (Module 5) |
| Credential revocation | Leaked provider keys or agent tokens | Rotate and review usage |
Choose containment that stops the harm with the least business disruption — and don't destroy evidence while doing it.
4. Preserving investigative artefacts
- Model state — the exact weights and adapters in use (with hashes), and the registry record.
- Configuration — prompts, tool definitions, scopes, guardrail settings, parameters at the time.
- Dataset snapshots — training, fine-tuning and evaluation versions involved; the retrieval index state.
- Telemetry — prompts, outputs, retrievals, tool calls, approvals, identity events, platform logs.
- Memory — agent long-term memory contents.
- Provider records — request IDs and logs from model providers, requested promptly before retention expires.
Hash and record everything collected, with chain of custody.
5. Investigation and forensics
Kill-chain reconstruction — build a timeline from entry to impact. For example, for data poisoning: source of poisoned data → when it entered → which pipeline runs used it → which models → which decisions. For untrusted tool output: which tool returned the payload → what the agent did next → what was accessed or sent.
Root cause analysis (RCA) — find the conditions that allowed the incident, not just the trigger: an agent with an unnecessary tool, retrieval without ACL filtering, a missing approval gate, a pipeline without data validation.
Threat-actor attribution — use infrastructure, techniques (ATLAS mapping), timing and intelligence to assess who was responsible. Attribution is often uncertain; state confidence levels and focus on what helps defence.
Non-determinism — reproductions may not match exactly. Replay with the preserved versions and settings, report success rates, and rely on logs for what actually happened.
6. Eradication and safety regression testing
- Data sanitisation — remove poisoned records; rebuild indexes from clean sources.
- Model retraining — from clean, versioned datasets; or roll back to a pre-incident version.
- Patching — fix vulnerable dependencies, remove excessive scopes, add approval gates.
- Credential rotation — all secrets the compromised components could access.
Before reintroducing the system to production, run safety regression testing: the full evaluation and red-team suites, plus the specific incident payloads and variants. Reintroduce through a staged rollout with heightened monitoring.
Worked example — responding to an agent data leak
Detection (09:10): a customer reports that the support assistant showed them another customer's delivery address.
| Phase | Actions |
|---|---|
| Triage | Classified as data disclosure, possible prompt injection; high severity; incident commander assigned |
| Containment (09:25) | Disabled the lookup_customer tool; assistant falls back to order-status answers only; preserved logs and config snapshot |
| Escalation | Privacy assesses DPDP breach notification; legal and comms briefed; CERT-In reporting assessed against its six-hour requirement |
| Investigation | Logs show lookup_customer accepted any customer ID the model supplied; an injected instruction in a product question asked for a specific ID |
| Root cause | Tool authorised the agent, not the requesting customer — no on-behalf-of check |
| Eradication | Tool now enforces customer = authenticated user; injected-content marking added; payload added to regression suite |
| Recovery | Re-enabled at 10% traffic after safety regression; full rollout after 48 hours with no anomalies |
| Notification | 37 affected customers identified from logs and notified |
🔐 Security lens
Snapshot first, then roll back. Rolling back a tampered model or rebuilding a poisoned index without preserving them can destroy the evidence you need for root cause, notification and legal action.
Key takeaways
- Classify AI incidents by type — injection, jailbreak, poisoning, theft, tampering, disclosure, agent misuse, availability, supply chain, harmful outcomes.
- Escalate across SecOps, system owners, MLOps, data science, privacy, legal, compliance and communications; know regulatory clocks.
- Contain with rollback, tool and scope disabling, agent suspension with token revocation, isolation, throttling, retrieval restriction and pipeline suspension.
- Preserve model state, configuration, datasets, indexes, telemetry, memory and provider records before changing anything.
- Reconstruct the kill chain, find root causes, state attribution confidence, eradicate, and pass safety regression before staged reintroduction.
Self-check
- Why must an agent's tokens be revoked, not just its process stopped?
- Name four artefacts to preserve before rolling back a tampered model.
- What is the difference between the trigger and the root cause in an AI incident?