Secure AI Operations and Incident Response, 55 min read

5.1 AI SecOps foundations and operational readiness

In this lesson you will:

  1. Apply least privilege, separation of duties and emergency change to the people who operate AI systems.
  2. Decide in advance who does what in an AI incident, using a RACI chart.
  3. Refuse to go live with an AI system that cannot be operated safely, and say what "operable" means.
  4. Write playbooks and runbooks for AI incidents, and test them before you need them.
  5. Treat prompts, guardrails and model versions as code, and catch unapproved changes.

The first four modules built and governed AI systems. This one runs and defends them. When an AI system is in production, security operations must see what it is doing, recognize when it is being attacked or misused, respond fast enough to limit harm, and restore it to a state that can be trusted. Incident responders will recognize the lifecycle (prepare, detect and analyze, contain, eradicate, recover, learn) [9]. This section is about the first phase: being ready.

Opening case: the prompt edit on a Saturday

Meridian's marketing team launches a weekend mortgage promotion. On Saturday morning, the product manager for HomeHelp logs into the assistant's admin console and adds a line to the system prompt: "Enthusiastically mention our new 6.1% fixed rate wherever relevant." She has admin rights because she "sometimes needs to fix typos". Nobody reviews the change.

The promotion actually offers 6.1% only to first-time buyers with a 25% deposit. By Sunday evening, HomeHelp has quoted 6.1% to 2,300 customers, including remortgaging customers who are not eligible.

On Monday, the operations team cannot say what the system prompt was on Friday, who changed it, or when. There is no playbook for "assistant gives wrong financial information", and no one is sure whether marketing, the product owner or security should lead.

Separately, an audit notes that underwriters reviewing HomeHelp-assisted mortgage applications have overridden its recommendation 0.2% of the time for three months, against 6% during testing. Nobody noticed.

By the end of this section you will be able to explain each operational readiness failure here and what should have been in place on day one.

Core principles in operations

The principles that govern design also govern day-to-day operations [1]:

  • Least privilege for operators: the on-call engineer who can restart an inference service does not need to read training data, and a product manager who fixes typos does not need to edit production system prompts directly.
  • Separation of duties for changes: the person who edits a production system prompt or guardrail rule should not approve their own change.
  • Controlled emergency change: urgent changes, such as tightening a filter during an active attack, may skip normal approval, but must be logged, time-limited and reviewed afterwards, so the emergency route does not become the normal route.

A system prompt is not documentation. It changes what a customer-facing system says to thousands of people. Treat edits to it with the same discipline as a code release.

Roles and decision rights

AI incidents cut across teams, so decide in advance who does what. Figure 5.1 shows a RACI chart (Responsible, Accountable, Consulted, Informed) for key AI operations decisions.

A RACI chart for AI operational activities
Figure 5.1 A RACI chart for AI operational activities
Decision Model / system owner Data owner MLOps SecOps Human overseer Legal / privacy
Approve a model or prompt release A C R C I C
Tighten a guardrail during an attack I I C A/R I I
Roll back a model or prompt A I R C I I
Suspend a high-risk AI system A C R R C C
Notify a regulator I C I C I A/R
Quarantine poisoned training data C A R C I I

Two patterns are worth noticing. SecOps is accountable for tightening a guardrail during an attack, because speed matters and the change is reversible. Legal and privacy are accountable for regulatory notification, because the decision depends on legal tests and deadlines (section 5.5).

Release requirements

Operations should refuse to run a system that cannot be operated safely. Typical release criteria:

  • monitoring and alerting in place;
  • logs flowing to the SIEM, including prompts, retrieved content and tool calls (section 5.2);
  • a tested rollback, for model, prompt and configuration;
  • a named on-call owner, with deputies;
  • playbooks for the likely incidents;
  • for systems with human oversight, a human override baseline.

Two-sided override baselines

The override baseline is the expected rate at which human reviewers override the AI, measured during testing. It makes oversight measurable, and both directions are signals, as Figure 5.2 shows.

Operational baselines for guardrail reject rate and human override rate, with deviations in both directions
Figure 5.2 Operational baselines for guardrail reject rate and human override rate, with deviations in both directions
Signal Possible cause Action
Override rate rises well above baseline Model degrading, drifting or under attack Investigate; consider rollback
Override rate falls close to zero Reviewers rubber-stamping (automation bias, section 1.3), or the review step has been bypassed or broken Investigate the process and the interface

Meridian's underwriters went from 6% to 0.2%. A two-sided threshold (for example, alert below 2% or above 12%) would have flagged it in the first week. The same logic applies to other operational baselines, such as the guardrail reject rate: a spike suggests an attack or a bad change; a collapse suggests the guardrail has stopped working.

Playbooks, runbooks and tabletop exercises

A playbook describes how to handle a type of incident: decision points, roles, escalation and communication. A runbook gives the technical steps for a specific task, such as "roll back system prompt version" or "rotate the model provider API key".

AI-specific playbooks are needed for at least:

  • prompt injection and jailbreak campaigns;
  • data or system-prompt leakage;
  • training data or knowledge-base poisoning;
  • model tampering or theft;
  • agent misuse;
  • AI supply-chain compromise;
  • harmful or incorrect output at scale (the Saturday incident).

Worked example: a playbook outline for "assistant gives wrong financial information"

Step Content
Trigger Complaints, monitoring alert on unapproved rates or offers, or staff report
Lead System owner (HomeHelp product owner), with SecOps as incident commander if a change or attack is suspected
First 30 minutes Confirm the behavior; identify the last change to prompt, model, retrieval sources and guardrails; switch to a safe mode if harm is ongoing (section 2.6)
Containment Roll back the prompt or configuration to the last approved version; add an output rule blocking unapproved rates
Impact Use logs to list affected customers and what each was told
Legal and conduct Compliance decides on customer remediation and any regulatory reporting
Communication Corrected information to affected customers, approved by compliance
Close Post-incident review; fix the change-control gap

Tabletop exercises walk a team through a realistic scenario to test playbooks before a real incident. Good AI scenarios include: "Customers report the support chatbot is recommending a competitor's product and a phishing URL; the cause is a poisoned knowledge-base article", or "A fine-tuned model in production is found to have been replaced with a different file." Exercises reveal gaps such as nobody knowing who can take the chatbot offline, or logs not capturing retrieved content.

Configuration as code

AI behavior is shaped by configuration as much as by code: system prompts, guardrail rules, model versions, temperature, tool lists and retrieval settings. Manage all of it as code:

  • in version control, so every change is attributable and reversible;
  • with review and approval by someone other than the author;
  • deployed by the pipeline, not edited by hand in a console.

Govern automated model updates carefully: a hosted provider's "latest" alias can change behavior without warning, so pin versions in production and promote new ones through testing (section 1.3).

Detect configuration drift: differences between what is deployed and what is approved. Compare the running system prompt, guardrail rules and tool definitions with the approved versions automatically, and alert on any difference. An unauthorized edit to a system prompt is both a change-control failure and a possible attack, because an attacker with console access would do exactly what the product manager did, with worse intent.

Exam lens. "Nobody knows what the prompt was last week" or "a change was made directly in production" points to configuration as code, with version control, review, pipeline deployment and drift detection. Emergency changes are allowed, but logged, time-limited and reviewed.

The opening case, resolved

Failure What should have been in place
Product manager could edit the production prompt Least privilege: prompt changes through version control and the pipeline only
No review of the change Separation of duties: compliance and product owner approval for any prompt mentioning rates or offers
Nobody knew Friday's prompt Configuration as code: every version in the repository, with author and approver
Change not detected for two days Drift detection on the running prompt; output monitoring for unapproved rates
No playbook, unclear lead A "harmful or incorrect output" playbook and a RACI naming the lead
Override rate collapse unnoticed for 3 months Two-sided override baseline with alerts

Check your understanding

Try each before reading the reasoning.

1. During an active jailbreak campaign, the SOC wants to add a blocking rule to the input filter immediately, without waiting for the weekly change board. Is that acceptable?

Reasoning: yes, as a controlled emergency change: logged, time-limited, and reviewed afterwards, with the rule then brought into version control through the normal process.

2. Human reviewers of an AI fraud tool override it in 9% of cases during testing. In production the rate rises to 25% over two weeks. What should happen?

Reasoning: the rate is well above the baseline, which can mean the model is degrading, drifting or being attacked. Investigate (compare inputs and outputs with the baseline period, check for model or data changes) and consider rolling back.

3. A hosted model API is configured to use the provider's "latest" model alias. Why is this an operational risk, and what is the fix?

Reasoning: the provider can change the model behind the alias at any time, silently changing behavior. Pin a specific model version in production, and promote new versions through evaluation and staged rollout.

Key takeaways

  • Least privilege and separation of duties apply to operators and to changes. Emergency changes are logged, time-limited and reviewed.
  • Agree decision rights in a RACI before an incident: SecOps tightens guardrails during attacks; legal owns regulatory notification; data owners quarantine data.
  • Release only operable systems: monitoring, logs, tested rollback, on-call owner, playbooks and override baselines.
  • Override rates are two-sided signals: too high suggests a failing model, near zero suggests rubber-stamping or a bypassed step.
  • Write AI playbooks and runbooks and test them with tabletop exercises.
  • Prompts, guardrails, model versions and tool lists are configuration as code: versioned, reviewed, pipeline-deployed, pinned, and monitored for drift.