AI Governance, Ethics, Risk and Compliance, 75 min read

2.3 AI risk management and assessments

In this lesson you will:

  1. Apply the four NIST AI RMF functions to a real AI system.
  2. Write an AI risk as a complete scenario, and record it in a register that someone can act on.
  3. Analyze AI risk qualitatively and quantitatively, and decide whether a safeguard is worth its cost.
  4. Choose the right impact assessment (DPIA, FRIA or AI system impact assessment) for a scenario.
  5. Use blast radius and autonomy to place a system in a risk tier.
  6. Choose a risk treatment, record an acceptance properly, and define when to reassess.

Opening case: two risk reviews of the same agent

Meridian's finance team wants to roll out PayFlow, an AI agent that reads incoming supplier invoices, matches them to purchase orders, and pays them. You met its prototype in section 1.2, where an injected note in an invoice made it change a supplier's bank details.

Two reviews arrive at the AI governance committee.

Review A, from the project team: "Risk: AI. Likelihood: low. Impact: medium. Rating: amber. Control: the system prompt tells the agent not to change bank details. Owner: IT. Recommendation: accept."

Review B, from the security team: three pages, with five scenarios, a heat map, a cost estimate and a request for two design changes.

The committee chair asks: "Review A is shorter and says the risk is acceptable. Why should we fund Review B's changes?" By the end of this section you will be able to explain what is wrong with Review A, how to quantify PayFlow's main risk, and who has the authority to accept what is left.

The AI risk management cycle

AI risk management follows the same logic as any risk management (identify, analyze, evaluate, treat, monitor), applied to a wider set of harms. The NIST AI RMF organizes it into four functions, shown in Figure 2.12 [10].

The NIST AI RMF: Govern at the center, with Map, Measure and Manage, and the trustworthy AI characteristics
Figure 2.12 The NIST AI RMF: Govern at the center, with Map, Measure and Manage, and the trustworthy AI characteristics
  • Govern sets the culture, policies, roles, risk tolerance and accountability. It runs across everything else.
  • Map establishes context: what the system is for, who uses it, who is affected, what it depends on, and what could go wrong.
  • Measure analyzes, tests and tracks the risks with metrics and independent review.
  • Manage prioritizes and treats risks, responds to incidents and communicates.

The cycle repeats throughout the system's life, not just before launch.

Worked example: the four functions applied to PayFlow

Function What it means for PayFlow
Govern Finance owns PayFlow; the committee sets the appetite ("no autonomous payment changes"); the payments policy applies to the agent as it does to staff
Map PayFlow reads invoices from outside suppliers (untrusted content), can read the vendor master, and can schedule payments. Affected parties: Meridian, its suppliers, and auditors. Dependencies: a hosted LLM, the ERP system, the email gateway
Measure Red-team PayFlow with injected invoices; measure how often it proposes bank changes; track payment exceptions and overrides in production
Manage Remove the bank-change tool, add approval for payments above a threshold, monitor, and reassess when the model or tools change

Categories of AI risk

AI risks are broader than security risks. Figure 2.13 shows a practical grouping [1][3][11].

Six categories of AI risk with examples
Figure 2.13 Six categories of AI risk with examples
  • Security: prompt injection, poisoning, model theft, agent misuse, supply-chain compromise.
  • Privacy: memorization of personal data, re-identification, unlawful processing, cross-border transfers.
  • Performance and safety: hallucination, drift, unsafe advice, brittleness.
  • Fairness and rights: discrimination, decisions people cannot contest, opaque decisions.
  • Legal and compliance: AI and data protection law, intellectual property, contract breach.
  • Operational and strategic: vendor lock-in, cost overruns, outages, reputational damage.

Review A's risk statement was "AI". That is not a risk; it is a technology. A usable risk names what could happen, how, and with what consequence.

Writing a risk scenario

A good risk statement has five elements, shown in Figure 2.14: a threat source, a vector, the AI asset affected, the event, and the impact.

Anatomy of an AI risk scenario, with a control at each element
Figure 2.14 Anatomy of an AI risk scenario, with a control at each element

For PayFlow's main risk:

An external fraudster (threat source) embeds instructions in a supplier invoice (vector) that PayFlow processes (asset), causing it to change a supplier's bank details and schedule payment (event), diverting funds and creating a reportable incident (impact).

Writing the scenario this way is not bureaucracy. Each element is a place where a control can act: threat intelligence and monitoring on the source; treating document content as data on the vector; least privilege and approval gates on the asset; change detection on the event; payment controls and insurance on the impact.

The AI risk register

Each scenario is recorded in a risk register with enough detail to act on. A good AI entry includes:

  • the AI system (linked to its inventory entry) and the component affected (data, model, retrieval, tool, interface);
  • the lifecycle stage;
  • the risk scenario, written as above;
  • a threat reference where one exists (an OWASP LLM Top 10 entry or a MITRE ATLAS technique);
  • inherent likelihood and impact, existing controls, and residual rating;
  • the treatment decision, the risk owner, and the review date.

Compare that with Review A: no scenario, a control that is not a control (a prompt), and "IT" as owner of a finance risk.

Analyzing risk: qualitative and quantitative

Qualitative analysis rates likelihood and impact on scales (for example 1 to 5) and plots them on a heat map, as in Figure 2.15. It is fast, and suits risks that are hard to put a number on, such as fairness or reputational harm.

A five-by-five AI risk heat map with six plotted scenarios
Figure 2.15 A five-by-five AI risk heat map with six plotted scenarios

Quantitative analysis assigns money values, using formulas familiar from CISSP:

  • Single loss expectancy (SLE) = asset value × exposure factor: the loss from one occurrence.
  • Annualized rate of occurrence (ARO): how many times a year the event is expected.
  • Annualized loss expectancy (ALE) = SLE × ARO: the expected loss per year.
  • Value of a safeguard = ALE before − ALE after − annual cost of the safeguard.

Worked example: is the redesign worth it?

Meridian's finance and security teams estimate PayFlow's diversion risk:

Item Estimate Basis
Average diverted payment USD 85,000 Typical invoice in the target supplier segment
Exposure factor 60% Banks recover about 40% of diverted funds on average
Response, investigation and notification per incident USD 24,000 Prior payment-fraud incidents
SLE 85,000 × 0.6 + 24,000 = USD 75,000
ARO without changes 4 per year Red-team success rate × volume of external invoices × observed targeting
ALE before 75,000 × 4 = USD 300,000

The proposed safeguards are to remove the bank-change tool (changes go through the vendor-management process with call-back verification) and to require human approval for any payment to a supplier whose details changed in the last 90 days. Together they cost USD 40,000 a year in staff time and development, and are expected to cut the ARO to 0.2.

  • ALE after = 75,000 × 0.2 = USD 15,000.
  • Value of the safeguards = 300,000 − 15,000 − 40,000 = USD 245,000 a year.

The safeguards are clearly justified. This is the answer to the committee chair's question: Review B's changes are not a cost of caution; they are worth about a quarter of a million dollars a year.

Quantitative analysis is only as good as its estimates, and AI incidents have little history. In practice, organizations use quantitative analysis for the top-right cluster of the heat map, where the money is, and qualitative ratings for the rest.

Exam lens. Expect to calculate ALE and safeguard value. A safeguard is justified when the reduction in ALE exceeds its annual cost. Check your units: ARO is per year, so a "once every four years" event has an ARO of 0.25.

Impact assessments

Several kinds of assessment look at AI from different angles, and scenarios often ask which one applies.

Assessment Focus Typically required when
Data protection impact assessment (DPIA) Risks to individuals from processing personal data GDPR Article 35: processing likely to result in high risk, such as large-scale profiling [16]
Fundamental rights impact assessment (FRIA) Effects of a high-risk AI system on people's fundamental rights EU AI Act Article 27: certain deployers of high-risk systems, such as public bodies and some credit and insurance uses [14]
AI system impact assessment Effects of an AI system on individuals, groups and society ISO/IEC 42001 requirement; guidance in ISO/IEC 42005 [12][26]
Algorithmic impact assessment Public review of automated decision systems Some governments' policies for public-sector automated decisions

Worked example: which assessment for which Meridian system?

System Assessments needed Why
TalentMatch (CV ranking, EU applicants) DPIA; AI system impact assessment Large-scale profiling of personal data; high-risk AI. A FRIA is not mandatory for a private-sector employer, though it is good practice
Credit-limit model for EU consumers DPIA; FRIA; AI system impact assessment Creditworthiness is one of the uses for which deployers must do a FRIA
PayFlow AI system impact assessment; a DPIA only if it processes significant personal data (for example sole-trader suppliers) Mainly financial and operational risk

Blast radius, autonomy and risk tiering

Two ideas sharpen any assessment of agents. Blast radius is how much damage a system could do if it misbehaved or were compromised: the data it can reach, the actions it can take, the autonomy it has, and how far its effects extend outside the organization (Figure 2.16). Autonomous actions are actions taken without a human approving each one.

Blast radius grows with data reach, tools and actions, autonomy and external reach
Figure 2.16 Blast radius grows with data reach, tools and actions, autonomy and external reach

PayFlow's blast radius in its original design: it reads all supplier records (data), can change bank details and schedule payments (tools), does so without approval (autonomy), and moves money to external accounts (reach). Every ring is at its widest. The cheapest treatment is often to shrink a ring: narrower data scopes, fewer tools, approval gates, payment caps, isolation.

Risk tiering sorts uses so that effort is proportionate. Figure 2.17 uses two dimensions: impact on people and technical complexity and autonomy.

Risk tier grid of human impact against technical complexity and autonomy
Figure 2.17 Risk tier grid of human impact against technical complexity and autonomy
Tier Meridian example Required before go-live
1 (low) Internal meeting-summary tool on the approved enterprise assistant Inventory entry; acceptable use policy applies
2 (medium) HomeHelp answering product questions from public content Plus security review, guardrails, transparency notice, monitoring
3 (high) TalentMatch; the credit-limit model; PayFlow as originally designed Plus impact assessments, bias testing where relevant, human-in-the-loop, committee approval, legal review, independent validation

Two rules override the grid. Regulation sets the floor: anything in an EU AI Act high-risk category is always top tier. And design changes can move a system down: PayFlow without the bank-change tool, with payment approval, has a much smaller blast radius, though it remains tier 3 while it moves money.

Treating risk

The four treatment options are familiar, shown in Figure 2.18.

Four risk treatment options with AI examples
Figure 2.18 Four risk treatment options with AI examples
  • Mitigate: add controls to reduce likelihood or impact (permission-aware retrieval; human approval).
  • Transfer: shift the financial consequence through insurance or contract (vendor indemnity, cyber insurance). Accountability stays with the organization.
  • Avoid: stop or redesign the activity (PayFlow simply has no tool to change bank details).
  • Accept: consciously tolerate residual risk because it is within appetite.

Formal acceptance

Acceptance must be formal: documented, with a rationale, time-limited, signed by a person with the authority to accept that level of risk, and reviewed. That person is usually the business owner, escalating to the committee or executive level as severity rises. A security team should never accept business risk on the business's behalf, and IT should not own a finance process risk.

A proper acceptance record for PayFlow's residual risk might read:

Field Entry
Risk Residual diversion risk after redesign (ALE about USD 15,000)
Owner Group Financial Controller
Rationale Within finance's operational loss appetite; further controls would slow supplier payments materially
Conditions Bank-change tool removed; approval for changed-detail suppliers; monthly exception reporting
Expiry 12 months, or earlier on any reassessment trigger
Approved by AI governance committee, minute AIGC-2026-044

When to reassess

AI risk changes even when nobody changes the system. Define reassessment triggers in advance (Figure 2.19).

Events that should trigger reassessment of an AI system
Figure 2.19 Events that should trigger reassessment of an AI system

Common triggers: a new model version or provider update; new training data or fine-tuning; a new use, user group or country; new tools, data sources or permissions for an agent; drift or fairness alerts; an incident or near miss, here or at a peer; new threat intelligence such as a new jailbreak technique; new regulation; and a maximum interval by tier. Connect the triggers to change management, monitoring and threat intelligence so that reassessment happens automatically, not only on a calendar.

The opening case, resolved

What is wrong with Review A, line by line:

  • "Risk: AI" is not a risk. It names no threat, vector, event or impact.
  • "Likelihood: low" is unsupported. The prototype was fooled in testing, and PayFlow processes untrusted external content every day.
  • "Control: the system prompt" is not a control. Prompts are guidance, not enforcement (section 1.2).
  • "Owner: IT" is the wrong owner. The risk is to finance's payments, so finance owns it.
  • "Accept" is an acceptance without authority, rationale, conditions or expiry.

Review B's scenario-based analysis puts the main risk at an ALE of about USD 300,000, and shows that two design changes costing USD 40,000 a year reduce it to about USD 15,000. The residual risk can then be formally accepted by the Group Financial Controller, within appetite, with conditions and an expiry, and approved by the committee.

Check your understanding

Try each before reading the reasoning.

1. A chatbot data leak would cost USD 400,000 per incident and is expected once every two years. A control costing USD 90,000 a year would cut the rate to once every ten years. Is it worth it?

Reasoning: ALE before = 400,000 × 0.5 = 200,000. ALE after = 400,000 × 0.1 = 40,000. Value = 200,000 − 40,000 − 90,000 = USD 70,000 a year, so yes, the control is justified.

2. A project manager signs a risk acceptance for an AI model that will make lending decisions for EU consumers, saying "the residual risk is low". What is wrong?

Reasoning: lending decisions on creditworthiness are high-risk under the EU AI Act, so the system is top tier regardless of the manager's view. Acceptance at that level needs the business owner with appropriate authority, and committee approval, with a documented rationale, conditions and expiry, and it requires the DPIA and FRIA first.

3. A vendor updates the hosted model behind an approved customer assistant. Nothing in Meridian's code changes. Does the assistant need reassessment?

Reasoning: yes. A model change is a reassessment trigger: behavior, safety and accuracy can shift. Re-run the evaluation suite and the relevant risk checks before accepting the new version, and pin versions where possible.

Key takeaways

  • The NIST AI RMF's Govern, Map, Measure and Manage functions run across the whole life of a system.
  • AI risk covers security, privacy, performance and safety, fairness, legal and operational harms. "AI" is not a risk; a scenario is.
  • Write risks as threat source → vector → asset → event → impact, and record them with an owner and review date.
  • ALE = SLE × ARO. A safeguard is justified when the reduction in ALE exceeds its annual cost.
  • DPIA for personal data risk, FRIA for certain high-risk deployers, AI system impact assessment under ISO/IEC 42001.
  • Blast radius and autonomy drive tiering; regulation sets the floor; shrinking a ring is often the cheapest treatment.
  • Acceptance is formal, owned by the business, time-limited and conditional. Transfer moves cost, never accountability.
  • Define reassessment triggers in advance and wire them into change, monitoring and threat intelligence.