AI Security Operations, Incident Response & Recovery, 35 min read

Responding to, containing and investigating AI incidents

In this lesson you will:

  • Classify AI incidents with an AI-specific triage taxonomy
  • Escalate across SecOps, MLOps, data science, legal and compliance
  • Apply AI-specific containment, including rollback, isolation, rate limiting, pipeline suspension and agent termination
  • Preserve investigative artefacts such as model states, weights, configurations and dataset snapshots
  • Reconstruct attack kill chains, perform root cause analysis and consider attribution
  • Eradicate threats and run safety regression testing before reintroduction

The incident response lifecycle — prepare, detect and analyse, contain, eradicate, recover, learn — applies to AI. What differs is the evidence you need, the containment levers you have, and the people you must involve.

1. AI incident triage taxonomy

AI incident taxonomy with categories and examples.
Figure 8.9 An AI incident taxonomy. Classification drives the playbook, the responders and the notification duties.
Category Examples
Prompt injection Direct or indirect injection causing data disclosure or unauthorised actions
Jailbreak / safety bypass Prohibited or harmful content produced at scale
Data poisoning Manipulated training, fine-tuning or retrieval data
Model theft / extraction Weights exfiltrated; extraction campaign
Model tampering Unauthorised modification of weights, prompts or configuration
Data disclosure Personal or confidential data exposed via outputs, logs or retrieval
Agent misuse Unauthorised actions, excessive agency, lateral movement
Availability / abuse Resource exhaustion, denial of wallet
Supply-chain compromise Malicious model, adapter, package or MCP server
Harmful outcomes Discriminatory or unsafe decisions affecting people (may not be malicious)

2. Escalation paths

AI incidents are multi-disciplinary:

Escalation from SOC to MLOps, data science, privacy, legal, compliance and communications.
Figure 8.10 Multi-disciplinary escalation. The incident commander coordinates; each function owns its decisions.
  • SecOps / incident commander — coordinates the response.
  • System owner — decides on suspension, rollback and business continuity.
  • MLOps / platform — executes technical containment and recovery.
  • Data science — analyses model behaviour, identifies affected decisions, retrains.
  • Privacy and legal — assess breach notification duties and legal exposure.
  • Compliance — regulatory reporting (for example, serious-incident reporting duties for high-risk AI under the EU AI Act; in India, CERT-In directions require reporting of specified cyber incidents within six hours of noticing them, and the DPDP framework requires personal data breach notification).
  • Communications — customers, partners, media.

3. AI-specific containment

Action When Notes
Model rollback Tampering, poisoning, bad update To last signed, approved version
Disable a tool or scope Agent misuse, injection with actions Least disruptive way to remove impact
Suspend or terminate an agent Agent compromise or runaway behaviour Revoke its tokens too — stopping the process isn't enough
Endpoint isolation Exposed or compromised serving endpoint Remove from network or restrict to known callers
Rate limiting / throttling Extraction, abuse, denial of wallet Per identity or globally
Restrict retrieval Poisoned or leaking knowledge base Remove documents or sources from the index
Pipeline suspension Poisoned data or compromised pipeline Freeze training and promotion; prevent further contaminated models
Fallback mode Primary system untrustworthy Smaller model, rules, human queue (Module 5)
Credential revocation Leaked provider keys or agent tokens Rotate and review usage

Choose containment that stops the harm with the least business disruption — and don't destroy evidence while doing it.

4. Preserving investigative artefacts

AI forensic artefacts to preserve.
Figure 8.11 What to preserve. Snapshot before you roll back, rebuild or retrain.
  • Model state — the exact weights and adapters in use (with hashes), and the registry record.
  • Configuration — prompts, tool definitions, scopes, guardrail settings, parameters at the time.
  • Dataset snapshots — training, fine-tuning and evaluation versions involved; the retrieval index state.
  • Telemetry — prompts, outputs, retrievals, tool calls, approvals, identity events, platform logs.
  • Memory — agent long-term memory contents.
  • Provider records — request IDs and logs from model providers, requested promptly before retention expires.

Hash and record everything collected, with chain of custody.

5. Investigation and forensics

Kill-chain reconstruction — build a timeline from entry to impact. For example, for data poisoning: source of poisoned data → when it entered → which pipeline runs used it → which models → which decisions. For untrusted tool output: which tool returned the payload → what the agent did next → what was accessed or sent.

Root cause analysis (RCA) — find the conditions that allowed the incident, not just the trigger: an agent with an unnecessary tool, retrieval without ACL filtering, a missing approval gate, a pipeline without data validation.

Threat-actor attribution — use infrastructure, techniques (ATLAS mapping), timing and intelligence to assess who was responsible. Attribution is often uncertain; state confidence levels and focus on what helps defence.

Non-determinism — reproductions may not match exactly. Replay with the preserved versions and settings, report success rates, and rely on logs for what actually happened.

6. Eradication and safety regression testing

  • Data sanitisation — remove poisoned records; rebuild indexes from clean sources.
  • Model retraining — from clean, versioned datasets; or roll back to a pre-incident version.
  • Patching — fix vulnerable dependencies, remove excessive scopes, add approval gates.
  • Credential rotation — all secrets the compromised components could access.

Before reintroducing the system to production, run safety regression testing: the full evaluation and red-team suites, plus the specific incident payloads and variants. Reintroduce through a staged rollout with heightened monitoring.

Worked example — responding to an agent data leak

Detection (09:10): a customer reports that the support assistant showed them another customer's delivery address.

Phase Actions
Triage Classified as data disclosure, possible prompt injection; high severity; incident commander assigned
Containment (09:25) Disabled the lookup_customer tool; assistant falls back to order-status answers only; preserved logs and config snapshot
Escalation Privacy assesses DPDP breach notification; legal and comms briefed; CERT-In reporting assessed against its six-hour requirement
Investigation Logs show lookup_customer accepted any customer ID the model supplied; an injected instruction in a product question asked for a specific ID
Root cause Tool authorised the agent, not the requesting customer — no on-behalf-of check
Eradication Tool now enforces customer = authenticated user; injected-content marking added; payload added to regression suite
Recovery Re-enabled at 10% traffic after safety regression; full rollout after 48 hours with no anomalies
Notification 37 affected customers identified from logs and notified

🔐 Security lens

Snapshot first, then roll back. Rolling back a tampered model or rebuilding a poisoned index without preserving them can destroy the evidence you need for root cause, notification and legal action.

Key takeaways

  • Classify AI incidents by type — injection, jailbreak, poisoning, theft, tampering, disclosure, agent misuse, availability, supply chain, harmful outcomes.
  • Escalate across SecOps, system owners, MLOps, data science, privacy, legal, compliance and communications; know regulatory clocks.
  • Contain with rollback, tool and scope disabling, agent suspension with token revocation, isolation, throttling, retrieval restriction and pipeline suspension.
  • Preserve model state, configuration, datasets, indexes, telemetry, memory and provider records before changing anything.
  • Reconstruct the kill chain, find root causes, state attribution confidence, eradicate, and pass safety regression before staged reintroduction.

Self-check

  1. Why must an agent's tokens be revoked, not just its process stopped?
  2. Name four artefacts to preserve before rolling back a tampered model.
  3. What is the difference between the trigger and the root cause in an AI incident?