2.6 Accountability, human oversight and auditing
In this lesson you will:
- Apply the three lines model to AI, and spot when one line is doing another line's job.
- Assign the authority to intervene, override, suspend and sign off, and make that authority usable in practice.
- Design an escalation path and an emergency stop that works outside the model.
- Find shadow AI using several sources, and respond in a way that does not drive it underground.
- Keep the governance records each stage of the lifecycle needs, and label AI interactions and content.
- Plan an AI audit, and know what evidence to ask for first.
Opening case: the night nobody could stop the bot
On a Friday evening, HomeHelp starts telling customers that Meridian is "currently offering a 0.5% rate discount for customers who switch their mortgage this weekend". There is no such offer. A prompt injection planted in a public mortgage-comparison page, which HomeHelp's retrieval was allowed to read, is the cause.
The on-call engineer sees the complaints at 9 p.m. and wants to switch HomeHelp off. But the runbook says only the "system owner" can take it offline, and the system owner is listed as "Digital Channels". Nobody answers the shared mailbox. The engineer messages the CISO, who says she cannot authorize a business decision. The head of digital is on a flight. HomeHelp keeps running until 11:40 p.m. and answers about 1,400 questions in that time.
The following week, internal audit is asked to review what went wrong. The audit team discovers that the AI governance team had also designed HomeHelp's guardrails, and that the HomeHelp project team had approved its own model update a month earlier.
By the end of this section you will be able to identify every governance failure in this story and design the arrangements that would have let HomeHelp be stopped in minutes.
The three lines model for AI
Accountability starts with clear roles. The three lines model, familiar from enterprise risk management, applies directly to AI, as Figure 2.30 shows.
- First line (own and manage): business owners, product teams and data scientists. They build and run AI systems, carry out impact assessments, implement controls and monitor models. They own the risk.
- Second line (oversee): risk, security, privacy, compliance and the responsible AI function. They set policy and risk tiers, review high-risk uses, and challenge the first line.
- Third line (assure): internal audit. It independently tests whether the design and operation of the AI framework work, and reports to the board's audit committee.
The common failure is a line doing another line's job. In the opening case, the AI governance team (second line) designed HomeHelp's guardrails, which means it would later be reviewing its own work. And the project team (first line) approved its own model update, which means nobody independent challenged a change that could alter behavior. Both destroy the independence that makes the model work.
Exam lens. When a question asks who should do something, check independence. The function that builds a control should not be the one that approves it, and internal audit should never design the controls it will audit.
Oversight authority
Accountability means a named person answers for an AI system's outcomes, can explain how it is controlled, and has the authority to act. Section 1.3 introduced oversight modes (human-in-the-loop and human-on-the-loop). Governance must now decide who holds each kind of authority, and make sure they can actually use it [1][14]. Figure 2.31 shows a typical authority matrix.
| Authority | Meaning | Typically held by |
|---|---|---|
| Intervene | Pause or correct an individual output or action | Trained operator or reviewer |
| Override | Replace the AI's decision with a human decision | Reviewer with domain competence |
| Suspend (stop) | Take the whole system offline | Named system owner and the on-call incident commander in an emergency |
| Sign-off | Approve a model or use for production, or a major change | Business owner, with committee approval for high-risk |
The single most important design decision in that table is the emergency suspend right for the incident commander. In the opening case, only "Digital Channels" could stop HomeHelp, and "Digital Channels" was a mailbox. An emergency stop that requires a business executive on a Friday night is not an emergency stop.
Authority must also be usable. Reviewers need training and competence, enough time per case, information about the AI's confidence and reasoning, an interface that makes overriding easy, and no penalty for overriding. The EU AI Act requires high-risk systems to be designed so that overseers can understand the system's capabilities and limits, stay aware of automation bias, interpret outputs, decide not to use them, and interrupt the system through a stop button or similar procedure [14].
Escalation and the emergency stop
Every AI system above the lowest tier needs a defined escalation path:
- What counts as an issue: harmful or false output, a suspected attack, a fairness alert, unexplained behavior, a spike in complaints.
- Who is told first: the on-call operations or security lead, not a shared mailbox.
- How fast it moves up: for example, to the system owner within 30 minutes and to the incident commander immediately if customer harm is ongoing.
- Who decides to stop: named individuals, with deputies.
- What happens after: communication, investigation, fix, and a decision to restore.
The emergency stop (or kill switch) must work outside the model. A system prompt that says "stop answering" is not a stop. Real stops include disabling the endpoint, routing traffic to a fallback ("our assistant is unavailable; here is how to reach us"), revoking an agent's credentials, or switching off a retrieval source. They must be tested, like any disaster recovery procedure.
Worked example: HomeHelp's escalation runbook, rewritten
| Step | Before | After |
|---|---|---|
| Detection | Complaints noticed manually | Alert when answers mention offers or rates not in the product catalog |
| First responder | On-call engineer | On-call engineer, with authority to switch HomeHelp to fallback mode if customer harm is ongoing |
| Escalation | "Digital Channels" mailbox | Named product owner and deputy by phone; incident commander in parallel |
| Stop mechanism | Unclear | Feature flag routes all traffic to a static fallback page; tested monthly |
| Retrieval kill switch | None | External web sources can be disabled independently of the whole assistant |
| Record | None | Every stop logged: who, when, why, and the restore decision |
With these changes, the on-call engineer could have switched HomeHelp to fallback mode at 9:02 p.m. instead of 11:40 p.m., and the 1,400 misleading answers would have been about 20.
Every exercise of authority (an override, a stop, an approval) should be recorded with who, when, why and what happened next. Those records are evidence for auditors and regulators, and data for improving the system.
Shadow AI
Shadow AI is AI used without the organization's knowledge or approval: staff using personal accounts on public chatbots, browser extensions with AI features, AI features switched on inside approved SaaS products, or developers calling model APIs from code. Its main risks are leakage of confidential or personal data, unvetted vendors, and decisions made with tools nobody has assessed. TalentMatch in section 2.1 was shadow AI.
No single source sees all of it, so discovery combines several (Figure 2.32).
- Network and cloud access security broker (CASB) logs: traffic to known AI service domains and APIs.
- Endpoints: installed AI apps, browser extensions and locally run models.
- Identity: OAuth consents granted to AI apps, and new service principals.
- SaaS management: AI features enabled inside sanctioned applications.
- Finance: card payments and expense claims for AI subscriptions.
- Code and cloud: AI SDKs and API keys in repositories; GPU services in cloud accounts.
- People: surveys, AI champions and amnesty programs.
Worked example: a month of discovery at Meridian
| Signal | Finding | Response |
|---|---|---|
| CASB | 2,300 staff used public chatbots in a month; 40 pasted text matching customer account formats | Provide the approved enterprise assistant; DLP coaching messages on paste; block uploads of classified files to unapproved services |
| Identity | 85 OAuth grants to an AI meeting-notes app with access to calendars and mailboxes | Revoke; assess the app; offer the approved alternative |
| Finance | 12 corporate-card subscriptions to AI coding assistants | Route to intake; approve one enterprise option |
| Code scanning | 3 model API keys committed to repositories | Rotate keys; add secrets scanning to CI |
| SaaS management | The CRM vendor enabled an "AI email writer" by default | Turn off until assessed; register in the inventory |
The response matters as much as detection. Blanket bans tend to drive AI use underground, where it is harder to see. A better approach: provide approved alternatives that meet people's real needs (here, an enterprise assistant with data-protection terms), publish a clear acceptable use policy, block only the highest-risk services and behaviors, and bring discovered uses into the inventory and intake process. The cycle is discover → assess → sanction, replace or block → communicate.
Governance records
Good governance leaves a trail that lets anyone (an auditor, a regulator, a new team member) understand a system. Figure 2.33 shows the records each lifecycle stage produces.
- Model cards: intended use, out-of-scope uses, training data summary, performance across groups and conditions, limitations and ethical considerations [23].
- Datasheets for datasets: motivation, composition, collection, preprocessing, uses and maintenance of a dataset [24].
- System documentation: architecture, components, data flows, controls and dependencies (the EU AI Act's technical documentation for high-risk systems).
- Assessments: risk assessments, DPIAs, FRIAs and AI impact assessments, with approvals.
- Decision logs: significant design and approval decisions, and the reasons for them.
- Operational records: monitoring reports, override records, incident records and change records.
- Exception records: approved deviations from policy, with compensating controls, owner and expiry date.
- Retirement records: decommissioning, and evidence that data and model copies were disposed of.
These must be kept current. Documentation describing last year's model is a liability, not evidence.
Labeling AI interactions and synthetic content
Transparency duties increasingly require labeling. Under the EU AI Act, people must be told when they are interacting with an AI system unless it is obvious; providers of generative AI must mark synthetic content in a machine-readable way so it can be detected as AI-generated; and deployers of deepfakes must disclose that content is artificial [14].
Techniques include:
- visible labels and notices ("You are chatting with Meridian's AI assistant");
- metadata and content credentials, such as the C2PA standard, which attach a signed record of how content was made;
- watermarking, which embeds a signal in the content itself.
Each has weaknesses: metadata is easily stripped, and watermarks can be degraded by editing, compression or paraphrasing. Using more than one technique is common practice.
Auditing AI
An AI audit checks that the governance program and individual systems work as claimed. Internal audit provides assurance to management and the board; external audit provides independent assurance, for example ISO/IEC 42001 certification audits, or assessments required by regulators or customers. A complete AI audit program covers:
| Area | Example questions |
|---|---|
| Compliance | Is the system classified correctly? Are legal obligations met? |
| Governance | Is it in the inventory, approved, owned and documented? |
| Bias and fairness | Were fairness tests run? Do production outcomes stay within thresholds? |
| Performance and drift | Is performance monitored? Were drift alerts acted on? |
| Security | Were threat modeling, red teaming and access reviews done? Are logs complete? |
| Oversight | Do overseers have authority, time and information? Does the stop work? |
| Outcomes | Is the system achieving its purpose without unacceptable harm or complaints? |
Auditors look for evidence, not intentions: test reports, logs, approval records, dashboards and remediation tickets. When a question asks what an auditor should request first, the answer is usually the inventory and the documented risk assessment, because everything else hangs on them.
Worked example: the HomeHelp audit plan
| Area | Evidence requested | Finding |
|---|---|---|
| Governance | Inventory record; owner | Owner recorded as a team mailbox, not a person |
| Change | Approval record for last model update | Approved by the project team itself, with no second-line review |
| Independence | Who designed and who reviews the guardrails | Same team; independence compromised |
| Oversight | Authority matrix; stop procedure; test records | No emergency authority for on-call; stop never tested |
| Security | Threat model; retrieval sources | External web pages allowed as retrieval sources without review |
| Outcomes | Incident timeline; customer impact | 1,400 affected answers; 2 hours 40 minutes to stop |
The opening case, resolved
| Failure | Fix |
|---|---|
| Owner was a mailbox | A named owner and deputy in the inventory |
| Only the owner could stop the system | Emergency suspend authority for the on-call incident commander |
| No tested stop mechanism | Feature-flag fallback, tested monthly; a separate kill switch for retrieval sources |
| CISO said she could not authorize a business decision | Pre-agreed authority matrix, so stopping for ongoing customer harm is a pre-authorized security decision |
| Governance team designed guardrails it later reviews | Guardrail design sits in the first line; the second line reviews and challenges |
| Project team approved its own model update | Model changes for tier 2 and above need second-line review before release |
| Untrusted web pages fed retrieval | Approved retrieval sources only, reviewed in intake (Module 4 covers the technical controls) |
Check your understanding
Try each before reading the reasoning.
1. Meridian's data science team proposes that, to save time, it will validate its own credit models before release, and internal audit will check a sample afterwards. What is wrong?
Reasoning: validation should be independent of development (a second-line or separate first-line validation function). Internal audit is the third line; it assures the framework, it does not replace independent validation. Self-validation removes the challenge that catches errors.
2. A business unit wants to ban all public AI tools after a data leak. What is a better response?
Reasoning: provide an approved alternative that meets the need, apply DLP and blocking to the highest-risk behaviors (such as uploading classified data), publish an acceptable use policy, and bring discovered uses into intake. A blanket ban without alternatives drives use underground and reduces visibility.
3. An external auditor beginning an ISO/IEC 42001 audit asks for one thing first. What is it most likely to be, and why?
Reasoning: the AI inventory (with risk tiers and owners) and the risk assessment, because they define the scope of the management system and every other control traces back to them.
Key takeaways
- The three lines (own, oversee, assure) only work with independence. Builders don't approve their own work; audit doesn't design controls.
- Assign intervene, override, suspend and sign-off authority to named roles, and give the incident commander an emergency stop.
- Make oversight usable: training, time, information, an easy override and no penalty for using it.
- Emergency stops work outside the model (feature flags, fallbacks, credential revocation) and are tested.
- Find shadow AI from several sources; respond with approved alternatives and intake, not blanket bans.
- Keep current records at every stage: model cards, datasheets, assessments, decision logs, overrides, exceptions with expiry, and retirement evidence.
- Label AI interactions and synthetic content with more than one technique.
- Auditors want evidence, and they start with the inventory and the risk assessment.