AI Security Operations, Incident Response & Recovery, 25 min read

AI SecOps foundations and operational readiness

In this lesson you will:

  • Apply least privilege and separation of duties to AI operations
  • Define AI operational roles and responsibilities
  • Set release requirements such as human-override baselines and reject-rate thresholds
  • Prepare playbooks, runbooks and tabletop exercises for AI incidents
  • Manage configuration, automated model updates and configuration drift

Security operations for AI starts before the first alert. Teams need clear roles, release criteria that define "normal", rehearsed playbooks and control over configuration — because in AI, configuration is behaviour.

1. Core principles in AI operations

  • Least privilege — operators get only the access needed: viewing telemetry doesn't require changing prompts; changing guardrails doesn't require downloading weights.
  • Separation of duties — the person who changes a model, prompt or guardrail is not the person who approves it; incident responders who can roll back models are distinct from developers who can deploy new ones, with break-glass procedures for emergencies.
  • Need to know for telemetry — prompt and output logs contain sensitive content; access is restricted and logged.

2. Roles and responsibilities

RACI chart for AI operational activities.
Figure 8.1 Who does what in AI operations. Clarity before an incident saves hours during one.
Role Responsibility
Model / system owner Accountable for the AI system in production; decides on rollback, suspension and risk acceptance
Data owner Accountable for data used in training and retrieval; decides on data quarantine and purges
MLOps / platform engineers Operate pipelines, registries, serving infrastructure; execute rollbacks and redeployments
SOC analysts Monitor AI telemetry, triage alerts, run playbooks
AI security engineers Detection engineering for AI, threat modelling, red-team liaison
Data scientists Diagnose model behaviour, retrain, evaluate fixes
Privacy, legal, compliance Breach assessment, regulatory notification, evidence handling
Communications Customer and public statements

3. Release requirements as operational baselines

Release criteria (Module 5) double as the baseline that operations monitors against:

  • Quality thresholds — accuracy, groundedness and task success rates.
  • Safety thresholds — jailbreak-suite pass rate; harmful-output rate.
  • Reject-rate thresholds — the expected rate at which guardrails block inputs or outputs. A sudden rise can mean an attack (or a bad change); a sudden fall can mean a guardrail stopped working.
  • Human-override baselines — the expected rate at which reviewers override AI decisions in HITL processes. Large deviations in either direction need investigation.
  • Latency, cost and volume baselines — per identity and in aggregate.
Operational baselines with normal bands and alert thresholds.
Figure 8.2 Baselines with normal bands. Deviations in either direction are signals.

4. Playbooks, runbooks and tabletop exercises

  • Playbooks describe the response to an incident type: triggers, roles, decisions, escalation, communications.
  • Runbooks are the technical steps: how to roll back a model, disable a tool, rotate a provider key, rebuild a vector index, suspend an agent.

AI incident types that deserve their own playbooks:

Playbook Key decisions
Prompt injection / data exfiltration via assistant Disable affected tools; restrict retrieval; notify privacy
Data poisoning Quarantine data source; identify affected models via lineage; roll back
Model tampering or theft Verify artefact integrity; revoke access; assess IP and legal impact
Agent misuse / unauthorised actions Suspend agent; revoke tokens; reverse actions
Harmful or biased outputs at scale Restrict or suspend; communications; regulatory assessment
Provider incident or model change Switch to fallback; pin versions
Denial of wallet / resource abuse Throttle; block identities; enforce caps

Tabletop exercises rehearse these with the real people. Good AI scenarios: "Monday morning, the fraud model's approval rate for one region has doubled since Friday's retrain" (poisoning or bad data?); "A customer posts screenshots of your assistant revealing another customer's address" (injection or ACL failure?).

5. Configuration management and drift

AI behaviour is defined by configuration as much as by code: model versions, prompts, temperatures, tool lists, scopes, guardrail thresholds, retrieval sources.

  • Store all of it as code in version control; deploy through pipelines; review changes.
  • Automated model updates — whether a provider's silent update or your own scheduled retrain — go through the same gates and staged rollout. Pin versions where possible.
  • Configuration drift tracking — compare running configuration against the approved baseline continuously; alert on differences (a guardrail threshold changed in the console, a tool added to an agent outside the pipeline).

Worked example — a tabletop exercise on model tampering

Scenario: the SOC sees the production credit model's hash differs from the registry's approved hash after a weekend maintenance window.

Inject Expected decision
Hash mismatch alert Who owns this? (system owner) — invoke model-tampering playbook
Approval rates unchanged so far Contain anyway: roll back to the signed approved version; preserve the suspect artefact
Maintenance account used outside its change ticket Disable account; preserve logs; involve insider-risk team
Decisions made with the suspect model: 3,400 Data scientists compare decisions; legal assesses customer impact

Gaps found: no runbook for preserving a model artefact forensically; break-glass rollback required a developer who was on leave. Both fixed within a month.

🔐 Security lens

A reject rate that drops to zero is as alarming as one that spikes — it can mean an attacker found a bypass, or a misconfiguration disabled the guardrail. Alert on both directions.

Key takeaways

  • Apply least privilege, separation of duties and need-to-know to AI operations and telemetry.
  • Define owners for systems and data, and responsibilities for MLOps, SOC, AI security, data science, privacy and communications.
  • Release thresholds become operational baselines: quality, safety, reject rates, override rates, latency, cost.
  • Prepare AI-specific playbooks and runbooks; rehearse them in tabletop exercises.
  • Manage AI configuration as code; gate automated updates; detect configuration drift.

Self-check

  1. Why is a sudden fall in guardrail reject rate a signal worth investigating?
  2. Give two AI incident types that need their own playbooks.
  3. What is configuration drift in an AI system?