5.1 AI SecOps foundations and operational readiness
In this lesson you will:
- Apply least privilege, separation of duties and emergency change to the people who operate AI systems.
- Decide in advance who does what in an AI incident, using a RACI chart.
- Refuse to go live with an AI system that cannot be operated safely, and say what "operable" means.
- Write playbooks and runbooks for AI incidents, and test them before you need them.
- Treat prompts, guardrails and model versions as code, and catch unapproved changes.
The first four modules built and governed AI systems. This one runs and defends them. When an AI system is in production, security operations must see what it is doing, recognize when it is being attacked or misused, respond fast enough to limit harm, and restore it to a state that can be trusted. Incident responders will recognize the lifecycle (prepare, detect and analyze, contain, eradicate, recover, learn) [9]. This section is about the first phase: being ready.
Opening case: the prompt edit on a Saturday
Meridian's marketing team launches a weekend mortgage promotion. On Saturday morning, the product manager for HomeHelp logs into the assistant's admin console and adds a line to the system prompt: "Enthusiastically mention our new 6.1% fixed rate wherever relevant." She has admin rights because she "sometimes needs to fix typos". Nobody reviews the change.
The promotion actually offers 6.1% only to first-time buyers with a 25% deposit. By Sunday evening, HomeHelp has quoted 6.1% to 2,300 customers, including remortgaging customers who are not eligible.
On Monday, the operations team cannot say what the system prompt was on Friday, who changed it, or when. There is no playbook for "assistant gives wrong financial information", and no one is sure whether marketing, the product owner or security should lead.
Separately, an audit notes that underwriters reviewing HomeHelp-assisted mortgage applications have overridden its recommendation 0.2% of the time for three months, against 6% during testing. Nobody noticed.
By the end of this section you will be able to explain each operational readiness failure here and what should have been in place on day one.
Core principles in operations
The principles that govern design also govern day-to-day operations [1]:
- Least privilege for operators: the on-call engineer who can restart an inference service does not need to read training data, and a product manager who fixes typos does not need to edit production system prompts directly.
- Separation of duties for changes: the person who edits a production system prompt or guardrail rule should not approve their own change.
- Controlled emergency change: urgent changes, such as tightening a filter during an active attack, may skip normal approval, but must be logged, time-limited and reviewed afterwards, so the emergency route does not become the normal route.
A system prompt is not documentation. It changes what a customer-facing system says to thousands of people. Treat edits to it with the same discipline as a code release.
Roles and decision rights
AI incidents cut across teams, so decide in advance who does what. Figure 5.1 shows a RACI chart (Responsible, Accountable, Consulted, Informed) for key AI operations decisions.
| Decision | Model / system owner | Data owner | MLOps | SecOps | Human overseer | Legal / privacy |
|---|---|---|---|---|---|---|
| Approve a model or prompt release | A | C | R | C | I | C |
| Tighten a guardrail during an attack | I | I | C | A/R | I | I |
| Roll back a model or prompt | A | I | R | C | I | I |
| Suspend a high-risk AI system | A | C | R | R | C | C |
| Notify a regulator | I | C | I | C | I | A/R |
| Quarantine poisoned training data | C | A | R | C | I | I |
Two patterns are worth noticing. SecOps is accountable for tightening a guardrail during an attack, because speed matters and the change is reversible. Legal and privacy are accountable for regulatory notification, because the decision depends on legal tests and deadlines (section 5.5).
Release requirements
Operations should refuse to run a system that cannot be operated safely. Typical release criteria:
- monitoring and alerting in place;
- logs flowing to the SIEM, including prompts, retrieved content and tool calls (section 5.2);
- a tested rollback, for model, prompt and configuration;
- a named on-call owner, with deputies;
- playbooks for the likely incidents;
- for systems with human oversight, a human override baseline.
Two-sided override baselines
The override baseline is the expected rate at which human reviewers override the AI, measured during testing. It makes oversight measurable, and both directions are signals, as Figure 5.2 shows.
| Signal | Possible cause | Action |
|---|---|---|
| Override rate rises well above baseline | Model degrading, drifting or under attack | Investigate; consider rollback |
| Override rate falls close to zero | Reviewers rubber-stamping (automation bias, section 1.3), or the review step has been bypassed or broken | Investigate the process and the interface |
Meridian's underwriters went from 6% to 0.2%. A two-sided threshold (for example, alert below 2% or above 12%) would have flagged it in the first week. The same logic applies to other operational baselines, such as the guardrail reject rate: a spike suggests an attack or a bad change; a collapse suggests the guardrail has stopped working.
Playbooks, runbooks and tabletop exercises
A playbook describes how to handle a type of incident: decision points, roles, escalation and communication. A runbook gives the technical steps for a specific task, such as "roll back system prompt version" or "rotate the model provider API key".
AI-specific playbooks are needed for at least:
- prompt injection and jailbreak campaigns;
- data or system-prompt leakage;
- training data or knowledge-base poisoning;
- model tampering or theft;
- agent misuse;
- AI supply-chain compromise;
- harmful or incorrect output at scale (the Saturday incident).
Worked example: a playbook outline for "assistant gives wrong financial information"
| Step | Content |
|---|---|
| Trigger | Complaints, monitoring alert on unapproved rates or offers, or staff report |
| Lead | System owner (HomeHelp product owner), with SecOps as incident commander if a change or attack is suspected |
| First 30 minutes | Confirm the behavior; identify the last change to prompt, model, retrieval sources and guardrails; switch to a safe mode if harm is ongoing (section 2.6) |
| Containment | Roll back the prompt or configuration to the last approved version; add an output rule blocking unapproved rates |
| Impact | Use logs to list affected customers and what each was told |
| Legal and conduct | Compliance decides on customer remediation and any regulatory reporting |
| Communication | Corrected information to affected customers, approved by compliance |
| Close | Post-incident review; fix the change-control gap |
Tabletop exercises walk a team through a realistic scenario to test playbooks before a real incident. Good AI scenarios include: "Customers report the support chatbot is recommending a competitor's product and a phishing URL; the cause is a poisoned knowledge-base article", or "A fine-tuned model in production is found to have been replaced with a different file." Exercises reveal gaps such as nobody knowing who can take the chatbot offline, or logs not capturing retrieved content.
Configuration as code
AI behavior is shaped by configuration as much as by code: system prompts, guardrail rules, model versions, temperature, tool lists and retrieval settings. Manage all of it as code:
- in version control, so every change is attributable and reversible;
- with review and approval by someone other than the author;
- deployed by the pipeline, not edited by hand in a console.
Govern automated model updates carefully: a hosted provider's "latest" alias can change behavior without warning, so pin versions in production and promote new ones through testing (section 1.3).
Detect configuration drift: differences between what is deployed and what is approved. Compare the running system prompt, guardrail rules and tool definitions with the approved versions automatically, and alert on any difference. An unauthorized edit to a system prompt is both a change-control failure and a possible attack, because an attacker with console access would do exactly what the product manager did, with worse intent.
Exam lens. "Nobody knows what the prompt was last week" or "a change was made directly in production" points to configuration as code, with version control, review, pipeline deployment and drift detection. Emergency changes are allowed, but logged, time-limited and reviewed.
The opening case, resolved
| Failure | What should have been in place |
|---|---|
| Product manager could edit the production prompt | Least privilege: prompt changes through version control and the pipeline only |
| No review of the change | Separation of duties: compliance and product owner approval for any prompt mentioning rates or offers |
| Nobody knew Friday's prompt | Configuration as code: every version in the repository, with author and approver |
| Change not detected for two days | Drift detection on the running prompt; output monitoring for unapproved rates |
| No playbook, unclear lead | A "harmful or incorrect output" playbook and a RACI naming the lead |
| Override rate collapse unnoticed for 3 months | Two-sided override baseline with alerts |
Check your understanding
Try each before reading the reasoning.
1. During an active jailbreak campaign, the SOC wants to add a blocking rule to the input filter immediately, without waiting for the weekly change board. Is that acceptable?
Reasoning: yes, as a controlled emergency change: logged, time-limited, and reviewed afterwards, with the rule then brought into version control through the normal process.
2. Human reviewers of an AI fraud tool override it in 9% of cases during testing. In production the rate rises to 25% over two weeks. What should happen?
Reasoning: the rate is well above the baseline, which can mean the model is degrading, drifting or being attacked. Investigate (compare inputs and outputs with the baseline period, check for model or data changes) and consider rolling back.
3. A hosted model API is configured to use the provider's "latest" model alias. Why is this an operational risk, and what is the fix?
Reasoning: the provider can change the model behind the alias at any time, silently changing behavior. Pin a specific model version in production, and promote new versions through evaluation and staged rollout.
Key takeaways
- Least privilege and separation of duties apply to operators and to changes. Emergency changes are logged, time-limited and reviewed.
- Agree decision rights in a RACI before an incident: SecOps tightens guardrails during attacks; legal owns regulatory notification; data owners quarantine data.
- Release only operable systems: monitoring, logs, tested rollback, on-call owner, playbooks and override baselines.
- Override rates are two-sided signals: too high suggests a failing model, near zero suggests rubber-stamping or a bypassed step.
- Write AI playbooks and runbooks and test them with tabletop exercises.
- Prompts, guardrails, model versions and tool lists are configuration as code: versioned, reviewed, pipeline-deployed, pinned, and monitored for drift.