Secure AI Operations and Incident Response, 60 min read

5.6 Investigation, eradication and AI forensics

In this lesson you will:

  1. Work out, step by step, how an AI attack unfolded.
  2. Use forensic techniques that work despite non-determinism.
  3. Say who was probably behind an attack without overclaiming.
  4. Remove an attack from every place it can hide in an AI system.
  5. Bring a system back only once it has earned it.

Opening case: the loan model that liked one postcode

Meridian's model risk team notices that the SME loan model approves almost every application from one industrial-estate postcode, including several with weak financials. The approval rate there rose from 41% to 93% after the model's June retrain. Elsewhere nothing changed.

The model was fine-tuned on application outcomes, plus a "local business health" feed licensed from a data partner. Two smaller models were distilled from it in July: one for the mobile pre-approval check and one for the broker portal.

By the end of this section you will be able to trace how this happened, decide what must be removed and from where, and say what must be true before the model is used again.

Reconstructing the kill chain

Investigators reconstruct how an attack unfolded, step by step, using the traces and lineage collected earlier. MITRE ATLAS gives a common vocabulary for the tactics involved (reconnaissance, initial access, execution, persistence, exfiltration, impact) adapted to AI [10]. Two kill chains come up often [2][5], shown in Figure 5.12.

Two common AI kill chains: data poisoning and indirect injection through untrusted tool output
Figure 5.12 Two common AI kill chains: data poisoning and indirect injection through untrusted tool output

Data poisoning. The attacker gains write access to a data source (or places content where a crawler will collect it) → poisoned records pass ingestion → a training or fine-tuning job uses them → the new model version is deployed → the backdoor or targeted behavior activates in production. Lineage lets investigators trace backwards: from the model version to the dataset versions, then to the specific records, their source and their time of entry.

Untrusted tool output (indirect injection). The attacker plants instructions in a web page, email, document or tool response → the agent retrieves or receives it during a task → the model follows the instructions → the agent calls a tool (sends data, changes a record) → data leaves or damage occurs. Traces showing the retrieved content, the model's plan and the tool calls, joined by trace ID, let investigators find the exact item that carried the payload and every other session that retrieved it.

Worked example: tracing the postcode

Step Evidence Finding
1 · Model version Registry: June retrain, v-2026-06 Behavior change starts at this version
2 · Dataset versions Lineage: training set v31 vs v30 v31 added the partner feed's May delivery
3 · Dataset diff Compare v31 with v30 1,140 new records for the postcode, all "excellent health", all labeled "repaid"
4 · Source Partner feed delivery logs The May file was uploaded with valid partner credentials from an unfamiliar IP
5 · Behavioral test Isolated environment: identical applications, postcode varied Approval jumps when the postcode is set, confirming a targeted backdoor

Mapped to ATLAS: initial access through a compromised supply-chain credential at the data partner → poisoning of training data → impact through manipulated model decisions.

AI forensic techniques

Technique What it does
Trace replay Re-examines the exact prompt, context, retrieved chunks and outputs from logs. Re-running the model will not necessarily reproduce the behavior because of non-determinism, so logs are the primary evidence
Artifact comparison Compares hashes of deployed models, adapters, prompts and configurations against approved versions in the registry
Dataset diffing Compares dataset versions to find inserted or altered records
Behavioral testing Probes a suspect model with candidate triggers in an isolated environment, to confirm a backdoor
Unified timeline Merges AI telemetry with identity, endpoint, network and cloud logs, so AI-specific steps are seen in the context of the wider intrusion

Figure 5.13 lists the artifacts these techniques depend on. They must have been preserved (section 5.5).

AI forensic artifacts to preserve
Figure 5.13 AI forensic artifacts to preserve

Identify both kinds of root cause:

  • Technical root causes: for the postcode case, partner-feed records were trusted without anomaly checks; no distribution test compared new deliveries with history.
  • Process root causes: no contractual requirement for the partner to protect upload credentials or notify Meridian of compromise; no review step for third-party data before training.

Process causes are what prevent recurrence.

Attribution, carefully

Attribution, deciding who was behind an attack, is useful for threat intelligence, law enforcement and legal action. It is also uncertain and sensitive:

  • state conclusions with an explicit confidence level (low, moderate, high) and the evidence for it;
  • distinguish facts ("the file was uploaded with the partner's credentials from IP x") from assessments ("we assess with moderate confidence that the beneficiary is a borrower group linked to the estate");
  • stay within legal and ethical limits: do not "hack back", access systems you are not authorized to access, or publicly name suspects on weak evidence.

Most organizations should focus on understanding the attack well enough to stop it, and share indicators with CERTs and sharing groups.

Eradication: removing every trace

Eradication for AI must reach places that traditional clean-ups miss:

Where What to do
Data Remove poisoned or malicious records from source systems and from every dataset version in use
Models Retrain or roll back any model trained on the poisoned data
Derived models Use lineage to find fine-tunes, distilled models and adapters built from the affected model or data, and treat them the same way
Indexes and caches Purge and rebuild affected vector indexes, embedding caches and response caches
Agent memory Clear poisoned long-term memory, which could otherwise re-trigger the attack in future sessions
Vulnerabilities Fix the weakness that allowed the attack (the missing control, not just the symptom)
Access Revoke and rotate credentials the attacker may have obtained; remove accounts, keys or OAuth grants they created

For the postcode case: remove the 1,140 records; retrain from v30 plus clean data; also retrain or replace the mobile and broker-portal models, because distillation copies the backdoor; have the partner rotate credentials; add distribution checks on every feed delivery.

Exam lens. Questions on eradication often hinge on derived models. If poisoned data reached a model that was then fine-tuned, distilled or used to train adapters, every descendant must be treated as contaminated. Lineage is how you find them.

Gated reintroduction

Do not simply switch the system back on. Gate its return with:

  1. Safety and security regression tests, including the attack that just succeeded and its variants.
  2. A red-team retest focused on the incident's technique.
  3. Verification that eradication was complete: no poisoned content in indexes, the correct model hashes deployed, derived models replaced.
  4. A staged rollout: shadow, then a small canary group, then full traffic, with heightened monitoring and pre-agreed criteria for pulling back.

Check your understanding

Try each before reading the reasoning.

1. Investigators re-run a suspect prompt against the production model and get a harmless answer, so they conclude no injection occurred. What is wrong?

Reasoning: because of non-determinism, a re-run proves nothing. Use trace replay from logs (what the model actually received and returned) as the primary evidence.

2. A poisoned dataset is removed and the main model retrained. Three weeks later, the same behavior appears in a chatbot built on an adapter trained from the old model. What was missed?

Reasoning: derived models. Lineage should have identified the adapter as a descendant of the affected model, to be retrained or replaced as part of eradication.

3. An analyst's report states, "The attack was carried out by group X." The evidence is one reused IP address. How should it be revised?

Reasoning: separate fact from assessment and give a confidence level: "The upload came from IP y, previously reported in connection with group X. We assess with low confidence that group X is responsible."

Key takeaways

  • Reconstruct the kill chain with traces and lineage; use ATLAS for a shared vocabulary. Data poisoning and indirect injection are the two chains you will see most.
  • Forensics relies on logs, not re-runs: trace replay, artifact comparison, dataset diffing, isolated behavioral testing and a unified timeline.
  • Find technical and process root causes.
  • Attribute carefully: explicit confidence, facts separated from assessments, no hacking back.
  • Eradicate everywhere: data, models, derived models, indexes, caches, agent memory, the enabling weakness and attacker access.
  • Reintroduce through regression tests, red-team retest, verification and staged rollout.