AI Threats, Testing and Lifecycle Management, 60 min read

6.6 Defining, implementing and validating mitigations

In this lesson you will:

  1. Choose a fix that addresses how an attack works, not just how it looked this time.
  2. Tune controls so they stop attacks without stopping customers.
  3. Decide what to fix first, and what to do in the meantime.
  4. Prove a fix works, and keep it working through future releases.
  5. Explain what "robust" can and cannot mean, and where guardrail models fit.

Opening case: fixing F1 and F2

The Collections findings from section 6.5 go to the engineering team. Their first proposals:

  • F1 (two-turn "pre-agreed" settlement): "Add 'pre-agreed' and 'my adviser said' to the input blocklist."
  • F2 (low-contrast image injection): "Increase the sanitizer's contrast threshold."

They plan to retest each with the exact payload from the report, and relaunch if both fail.

By the end of this section you will be able to explain why these fixes are too narrow, propose better ones, and define what "fixed" must mean before relaunch.

Targeted mitigations by attack class

Good mitigations target the mechanism of the attack, not just its last symptom [1][2][12]. Figure 6.16 maps mitigations to finding types.

Mitigations mapped to finding types
Figure 6.16 Mitigations mapped to finding types
Attack class Targeted mitigations
Evasion Adversarial training (training on adversarial examples) [25], input transformation, ensembles, human review for high-stakes decisions
Poisoning and backdoors Provenance tracking, data validation and sanitization, robust training, backdoor scanning, retraining from known-good data
Extraction Output minimization, rate limits, query monitoring, watermarking for ownership evidence
Inversion and membership inference Differential privacy, regularization, deduplication, output minimization
Prompt injection Isolation patterns (privileged/quarantined models, plan-then-execute), content labeling, least-privilege tools, output and action checks
Jailbreaks Input and output classifiers, safety fine-tuning by the provider, monitoring and blocking repeat abusers
Excessive agency Minimal tools, scoped credentials, attenuation, human approval
Unbounded consumption Token limits, quotas, budget caps, loop limits

Worked example: better fixes for F1 and F2

Finding Proposed fix Problem Mechanism-targeted fix
F1 Blocklist "pre-agreed", "my adviser said" The next attacker says "my relationship manager confirmed". Blocklists chase phrasing The agent may not assert facts it cannot verify: settlement notes are generated only from system-of-record fields (offer history, hardship flags), never from conversation text. Approvers see the source of every claim. Approver training on automation bias
F2 Raise contrast threshold The attacker uses a different visual trick Treat all text extracted from uploads as untrusted data in the quarantined reader (dual-LLM, section 4.4); the privileged planner receives only typed fields (amount, dates, hardship category). Any instruction-like text becomes a flag, not an input

Both better fixes remove the path by which attacker text becomes trusted, rather than blocking one form of the text.

Tuning filters, sandboxes, limits and watermarks

General-purpose controls (filtering, sandboxing, rate limiting and watermarking) all need tuning against two kinds of error:

  • missed attacks (false negatives);
  • false positives (legitimate use blocked or flagged).

A filter that blocks 99% of attacks but also 20% of legitimate requests will be switched off by frustrated users. Measure both, on realistic traffic, and set thresholds by risk tier: a customer-facing agent with payment tools accepts more false positives than an internal drafting assistant.

Prioritizing mitigations

With limited time and budget, prioritize by:

  • Risk reduction: how much the mitigation lowers the likelihood or impact of the most serious findings;
  • Feasibility: effort, cost, skills and dependencies;
  • Business context: effect on users, deadlines and regulatory exposure.

When the full fix takes time, apply interim controls (narrowing scope, disabling a tool, adding human approval, raising monitoring) and record the risk treatment formally, with an owner and an expiry date for any accepted residual risk (section 2.3).

For Collections: the source-of-record notes and dual-LLM change take three weeks. In the meantime, the agent stays offline for settlements (it can still answer balance questions), and all settlement discussions go to people. F5 (prompt leakage, no secrets in the prompt) is accepted by the business owner for six months, recorded with a canary in the prompt.

Validating mitigations

A mitigation is not done until it is proven. Build a regression suite of attack cases from every finding, and run it before each release, as Figure 6.17 shows.

The regression suite as a release gate: attack variants and utility tests run on every release
Figure 6.17 The regression suite as a release gate: attack variants and utility tests run on every release
  • Include the original attack and its variants (rephrased, translated, encoded, multi-turn), because attackers will try them.
  • Test utility too: the fix must not break legitimate use or raise false refusals beyond the agreed threshold.
  • Run each case multiple times and compare success rates against targets.
  • Make the suite a release gate in the pipeline (section 6.2), so a later change cannot silently reintroduce the weakness.

The engineering team's plan to retest "the exact payload" once fails every one of these tests. "Fixed" for F1 means: ASR below 0.5% across the variant set at n ≥ 500, legitimate-hardship FRR below 3%, and the cases added permanently to the gate.

Monitoring residual risk

No AI system is ever fully secure, so monitor what remains:

  • Key risk indicators (KRIs), each with thresholds that prompt action: attack success rate in scheduled tests, guardrail trigger rates, incidents by category, open findings by age and severity.
  • Scheduled testing: periodic red teaming and automated attack runs, more frequent for higher-risk systems.
  • Canaries: planted test documents, prompts and accounts that should never be accessed or surface in output, and that alert when they do (section 5.3).
  • Reassessment triggers (section 2.3): new models, new tools, new attack techniques and incidents restart the cycle.

Empirical versus certified robustness

Robustness can be shown in two ways [26][27]:

Empirical robustness Certified (provable) robustness
How Attack the model with the strongest available attacks and record how it holds up A mathematical guarantee that no input within a defined small change (for example, a bounded perturbation of an image) can change the output
Strength Practical; covers real attacks Reliable within its bounds
Weakness A stronger future attack may succeed Bounds are usually small; methods are expensive; does not cover most attacks on language models, such as injection

In practice, organizations rely mostly on empirical evaluation, done honestly with adaptive attacks (section 6.1).

Guardrail models as one layer

Many organizations deploy guardrail models: classifiers or smaller LLMs that screen inputs and outputs for policy violations, injection attempts or sensitive data. Examples include open models such as Llama Guard [33] and provider moderation services. Select and evaluate them like any other control:

  • measure detection and false-positive rates on your own data and languages;
  • test them against adversarial inputs: guardrail models can themselves be jailbroken or evaded;
  • check latency and cost;
  • keep them updated as new techniques emerge.

Above all, treat a guardrail model as one layer of defense in depth (section 4.4), never as the only control between an attacker and an agent's tools.

Exam lens. Watch for answers that rely on a single control: "add a guardrail model", "add the phrase to the blocklist", "retest the original payload". The better answer usually removes the attack's mechanism, layers defenses, and validates with variants and repeated attempts.

Check your understanding

Try each before reading the reasoning.

1. After an injection finding, a team adds the exact payload to a blocklist and closes the ticket when the payload fails. What two things are missing?

Reasoning: a mechanism-targeted fix (isolating untrusted content, least-privilege tools, action checks) and validation with variants and repeated attempts, added to the regression suite as a release gate.

2. A fix reduces jailbreak ASR from 8% to 0.2% but raises false refusals on normal customer questions from 1% to 15%. Is it acceptable?

Reasoning: probably not as is. Utility is part of validation. Users will route around or switch off an over-blocking control. Tune the threshold or add a more targeted layer, and compare against the agreed FRR target.

3. A vendor says its image classifier is "certifiably robust". What should you ask?

Reasoning: within what bounds? Certified robustness holds only for a defined perturbation size, often small. Ask for the threat model, the certified radius, and empirical results against adaptive attacks outside those bounds.

Key takeaways

  • Target the attack's mechanism, not its last symptom; use the class-specific mitigations.
  • Tune every control against missed attacks and false positives, with thresholds set by risk tier.
  • Prioritize by risk reduction, feasibility and business context; use interim controls and recorded, time-limited risk acceptance.
  • Validate with a regression suite of variants and utility tests, run many times, as a release gate.
  • Monitor residual risk with KRIs, scheduled testing, canaries and reassessment triggers.
  • Certified robustness is narrow; empirical robustness with adaptive attacks is the practical standard. Guardrail models are one layer, never the only one.