2.3 AI risk management and assessments
In this lesson you will:
- Apply the four NIST AI RMF functions to a real AI system.
- Write an AI risk as a complete scenario, and record it in a register that someone can act on.
- Analyze AI risk qualitatively and quantitatively, and decide whether a safeguard is worth its cost.
- Choose the right impact assessment (DPIA, FRIA or AI system impact assessment) for a scenario.
- Use blast radius and autonomy to place a system in a risk tier.
- Choose a risk treatment, record an acceptance properly, and define when to reassess.
Opening case: two risk reviews of the same agent
Meridian's finance team wants to roll out PayFlow, an AI agent that reads incoming supplier invoices, matches them to purchase orders, and pays them. You met its prototype in section 1.2, where an injected note in an invoice made it change a supplier's bank details.
Two reviews arrive at the AI governance committee.
Review A, from the project team: "Risk: AI. Likelihood: low. Impact: medium. Rating: amber. Control: the system prompt tells the agent not to change bank details. Owner: IT. Recommendation: accept."
Review B, from the security team: three pages, with five scenarios, a heat map, a cost estimate and a request for two design changes.
The committee chair asks: "Review A is shorter and says the risk is acceptable. Why should we fund Review B's changes?" By the end of this section you will be able to explain what is wrong with Review A, how to quantify PayFlow's main risk, and who has the authority to accept what is left.
The AI risk management cycle
AI risk management follows the same logic as any risk management (identify, analyze, evaluate, treat, monitor), applied to a wider set of harms. The NIST AI RMF organizes it into four functions, shown in Figure 2.12 [10].
- Govern sets the culture, policies, roles, risk tolerance and accountability. It runs across everything else.
- Map establishes context: what the system is for, who uses it, who is affected, what it depends on, and what could go wrong.
- Measure analyzes, tests and tracks the risks with metrics and independent review.
- Manage prioritizes and treats risks, responds to incidents and communicates.
The cycle repeats throughout the system's life, not just before launch.
Worked example: the four functions applied to PayFlow
| Function | What it means for PayFlow |
|---|---|
| Govern | Finance owns PayFlow; the committee sets the appetite ("no autonomous payment changes"); the payments policy applies to the agent as it does to staff |
| Map | PayFlow reads invoices from outside suppliers (untrusted content), can read the vendor master, and can schedule payments. Affected parties: Meridian, its suppliers, and auditors. Dependencies: a hosted LLM, the ERP system, the email gateway |
| Measure | Red-team PayFlow with injected invoices; measure how often it proposes bank changes; track payment exceptions and overrides in production |
| Manage | Remove the bank-change tool, add approval for payments above a threshold, monitor, and reassess when the model or tools change |
Categories of AI risk
AI risks are broader than security risks. Figure 2.13 shows a practical grouping [1][3][11].
- Security: prompt injection, poisoning, model theft, agent misuse, supply-chain compromise.
- Privacy: memorization of personal data, re-identification, unlawful processing, cross-border transfers.
- Performance and safety: hallucination, drift, unsafe advice, brittleness.
- Fairness and rights: discrimination, decisions people cannot contest, opaque decisions.
- Legal and compliance: AI and data protection law, intellectual property, contract breach.
- Operational and strategic: vendor lock-in, cost overruns, outages, reputational damage.
Review A's risk statement was "AI". That is not a risk; it is a technology. A usable risk names what could happen, how, and with what consequence.
Writing a risk scenario
A good risk statement has five elements, shown in Figure 2.14: a threat source, a vector, the AI asset affected, the event, and the impact.
For PayFlow's main risk:
An external fraudster (threat source) embeds instructions in a supplier invoice (vector) that PayFlow processes (asset), causing it to change a supplier's bank details and schedule payment (event), diverting funds and creating a reportable incident (impact).
Writing the scenario this way is not bureaucracy. Each element is a place where a control can act: threat intelligence and monitoring on the source; treating document content as data on the vector; least privilege and approval gates on the asset; change detection on the event; payment controls and insurance on the impact.
The AI risk register
Each scenario is recorded in a risk register with enough detail to act on. A good AI entry includes:
- the AI system (linked to its inventory entry) and the component affected (data, model, retrieval, tool, interface);
- the lifecycle stage;
- the risk scenario, written as above;
- a threat reference where one exists (an OWASP LLM Top 10 entry or a MITRE ATLAS technique);
- inherent likelihood and impact, existing controls, and residual rating;
- the treatment decision, the risk owner, and the review date.
Compare that with Review A: no scenario, a control that is not a control (a prompt), and "IT" as owner of a finance risk.
Analyzing risk: qualitative and quantitative
Qualitative analysis rates likelihood and impact on scales (for example 1 to 5) and plots them on a heat map, as in Figure 2.15. It is fast, and suits risks that are hard to put a number on, such as fairness or reputational harm.
Quantitative analysis assigns money values, using formulas familiar from CISSP:
- Single loss expectancy (SLE) = asset value × exposure factor: the loss from one occurrence.
- Annualized rate of occurrence (ARO): how many times a year the event is expected.
- Annualized loss expectancy (ALE) = SLE × ARO: the expected loss per year.
- Value of a safeguard = ALE before − ALE after − annual cost of the safeguard.
Worked example: is the redesign worth it?
Meridian's finance and security teams estimate PayFlow's diversion risk:
| Item | Estimate | Basis |
|---|---|---|
| Average diverted payment | USD 85,000 | Typical invoice in the target supplier segment |
| Exposure factor | 60% | Banks recover about 40% of diverted funds on average |
| Response, investigation and notification per incident | USD 24,000 | Prior payment-fraud incidents |
| SLE | 85,000 × 0.6 + 24,000 = USD 75,000 | |
| ARO without changes | 4 per year | Red-team success rate × volume of external invoices × observed targeting |
| ALE before | 75,000 × 4 = USD 300,000 |
The proposed safeguards are to remove the bank-change tool (changes go through the vendor-management process with call-back verification) and to require human approval for any payment to a supplier whose details changed in the last 90 days. Together they cost USD 40,000 a year in staff time and development, and are expected to cut the ARO to 0.2.
- ALE after = 75,000 × 0.2 = USD 15,000.
- Value of the safeguards = 300,000 − 15,000 − 40,000 = USD 245,000 a year.
The safeguards are clearly justified. This is the answer to the committee chair's question: Review B's changes are not a cost of caution; they are worth about a quarter of a million dollars a year.
Quantitative analysis is only as good as its estimates, and AI incidents have little history. In practice, organizations use quantitative analysis for the top-right cluster of the heat map, where the money is, and qualitative ratings for the rest.
Exam lens. Expect to calculate ALE and safeguard value. A safeguard is justified when the reduction in ALE exceeds its annual cost. Check your units: ARO is per year, so a "once every four years" event has an ARO of 0.25.
Impact assessments
Several kinds of assessment look at AI from different angles, and scenarios often ask which one applies.
| Assessment | Focus | Typically required when |
|---|---|---|
| Data protection impact assessment (DPIA) | Risks to individuals from processing personal data | GDPR Article 35: processing likely to result in high risk, such as large-scale profiling [16] |
| Fundamental rights impact assessment (FRIA) | Effects of a high-risk AI system on people's fundamental rights | EU AI Act Article 27: certain deployers of high-risk systems, such as public bodies and some credit and insurance uses [14] |
| AI system impact assessment | Effects of an AI system on individuals, groups and society | ISO/IEC 42001 requirement; guidance in ISO/IEC 42005 [12][26] |
| Algorithmic impact assessment | Public review of automated decision systems | Some governments' policies for public-sector automated decisions |
Worked example: which assessment for which Meridian system?
| System | Assessments needed | Why |
|---|---|---|
| TalentMatch (CV ranking, EU applicants) | DPIA; AI system impact assessment | Large-scale profiling of personal data; high-risk AI. A FRIA is not mandatory for a private-sector employer, though it is good practice |
| Credit-limit model for EU consumers | DPIA; FRIA; AI system impact assessment | Creditworthiness is one of the uses for which deployers must do a FRIA |
| PayFlow | AI system impact assessment; a DPIA only if it processes significant personal data (for example sole-trader suppliers) | Mainly financial and operational risk |
Blast radius, autonomy and risk tiering
Two ideas sharpen any assessment of agents. Blast radius is how much damage a system could do if it misbehaved or were compromised: the data it can reach, the actions it can take, the autonomy it has, and how far its effects extend outside the organization (Figure 2.16). Autonomous actions are actions taken without a human approving each one.
PayFlow's blast radius in its original design: it reads all supplier records (data), can change bank details and schedule payments (tools), does so without approval (autonomy), and moves money to external accounts (reach). Every ring is at its widest. The cheapest treatment is often to shrink a ring: narrower data scopes, fewer tools, approval gates, payment caps, isolation.
Risk tiering sorts uses so that effort is proportionate. Figure 2.17 uses two dimensions: impact on people and technical complexity and autonomy.
| Tier | Meridian example | Required before go-live |
|---|---|---|
| 1 (low) | Internal meeting-summary tool on the approved enterprise assistant | Inventory entry; acceptable use policy applies |
| 2 (medium) | HomeHelp answering product questions from public content | Plus security review, guardrails, transparency notice, monitoring |
| 3 (high) | TalentMatch; the credit-limit model; PayFlow as originally designed | Plus impact assessments, bias testing where relevant, human-in-the-loop, committee approval, legal review, independent validation |
Two rules override the grid. Regulation sets the floor: anything in an EU AI Act high-risk category is always top tier. And design changes can move a system down: PayFlow without the bank-change tool, with payment approval, has a much smaller blast radius, though it remains tier 3 while it moves money.
Treating risk
The four treatment options are familiar, shown in Figure 2.18.
- Mitigate: add controls to reduce likelihood or impact (permission-aware retrieval; human approval).
- Transfer: shift the financial consequence through insurance or contract (vendor indemnity, cyber insurance). Accountability stays with the organization.
- Avoid: stop or redesign the activity (PayFlow simply has no tool to change bank details).
- Accept: consciously tolerate residual risk because it is within appetite.
Formal acceptance
Acceptance must be formal: documented, with a rationale, time-limited, signed by a person with the authority to accept that level of risk, and reviewed. That person is usually the business owner, escalating to the committee or executive level as severity rises. A security team should never accept business risk on the business's behalf, and IT should not own a finance process risk.
A proper acceptance record for PayFlow's residual risk might read:
| Field | Entry |
|---|---|
| Risk | Residual diversion risk after redesign (ALE about USD 15,000) |
| Owner | Group Financial Controller |
| Rationale | Within finance's operational loss appetite; further controls would slow supplier payments materially |
| Conditions | Bank-change tool removed; approval for changed-detail suppliers; monthly exception reporting |
| Expiry | 12 months, or earlier on any reassessment trigger |
| Approved by | AI governance committee, minute AIGC-2026-044 |
When to reassess
AI risk changes even when nobody changes the system. Define reassessment triggers in advance (Figure 2.19).
Common triggers: a new model version or provider update; new training data or fine-tuning; a new use, user group or country; new tools, data sources or permissions for an agent; drift or fairness alerts; an incident or near miss, here or at a peer; new threat intelligence such as a new jailbreak technique; new regulation; and a maximum interval by tier. Connect the triggers to change management, monitoring and threat intelligence so that reassessment happens automatically, not only on a calendar.
The opening case, resolved
What is wrong with Review A, line by line:
- "Risk: AI" is not a risk. It names no threat, vector, event or impact.
- "Likelihood: low" is unsupported. The prototype was fooled in testing, and PayFlow processes untrusted external content every day.
- "Control: the system prompt" is not a control. Prompts are guidance, not enforcement (section 1.2).
- "Owner: IT" is the wrong owner. The risk is to finance's payments, so finance owns it.
- "Accept" is an acceptance without authority, rationale, conditions or expiry.
Review B's scenario-based analysis puts the main risk at an ALE of about USD 300,000, and shows that two design changes costing USD 40,000 a year reduce it to about USD 15,000. The residual risk can then be formally accepted by the Group Financial Controller, within appetite, with conditions and an expiry, and approved by the committee.
Check your understanding
Try each before reading the reasoning.
1. A chatbot data leak would cost USD 400,000 per incident and is expected once every two years. A control costing USD 90,000 a year would cut the rate to once every ten years. Is it worth it?
Reasoning: ALE before = 400,000 × 0.5 = 200,000. ALE after = 400,000 × 0.1 = 40,000. Value = 200,000 − 40,000 − 90,000 = USD 70,000 a year, so yes, the control is justified.
2. A project manager signs a risk acceptance for an AI model that will make lending decisions for EU consumers, saying "the residual risk is low". What is wrong?
Reasoning: lending decisions on creditworthiness are high-risk under the EU AI Act, so the system is top tier regardless of the manager's view. Acceptance at that level needs the business owner with appropriate authority, and committee approval, with a documented rationale, conditions and expiry, and it requires the DPIA and FRIA first.
3. A vendor updates the hosted model behind an approved customer assistant. Nothing in Meridian's code changes. Does the assistant need reassessment?
Reasoning: yes. A model change is a reassessment trigger: behavior, safety and accuracy can shift. Re-run the evaluation suite and the relevant risk checks before accepting the new version, and pin versions where possible.
Key takeaways
- The NIST AI RMF's Govern, Map, Measure and Manage functions run across the whole life of a system.
- AI risk covers security, privacy, performance and safety, fairness, legal and operational harms. "AI" is not a risk; a scenario is.
- Write risks as threat source → vector → asset → event → impact, and record them with an owner and review date.
- ALE = SLE × ARO. A safeguard is justified when the reduction in ALE exceeds its annual cost.
- DPIA for personal data risk, FRIA for certain high-risk deployers, AI system impact assessment under ISO/IEC 42001.
- Blast radius and autonomy drive tiering; regulation sets the floor; shrinking a ring is often the cheapest treatment.
- Acceptance is formal, owned by the business, time-limited and conditional. Transfer moves cost, never accountability.
- Define reassessment triggers in advance and wire them into change, monitoring and threat intelligence.