1.3 AI limitations and performance
In this lesson you will:
- Explain why hallucination cannot be patched away, and design controls that limit its impact.
- Explain what non-determinism does to testing, incident investigation and evidence.
- Tell data, concept, label and behavioral drift apart, and say how each is detected.
- Explain model collapse and the provenance controls that prevent it.
- Calculate accuracy, precision, recall, F1 and the false positive rate from a confusion matrix, and explain why accuracy misleads in security.
- Set a decision threshold from the cost of each kind of error.
- Choose a human oversight mode for a decision, and tell meaningful oversight from rubber-stamping.
Opening case: the dashboard that stayed green
Eighteen months after launch, Meridian Bank's fraud model, Sentinel, has a dashboard on the wall of the operations center. It shows one number: accuracy 99.6%. It has shown roughly that number every week since launch, and the dashboard is green.
At the same time, Meridian's new mortgage assistant, HomeHelp, tells a customer that the bank "waives the early repayment fee for customers who have banked with us for over ten years". There is no such policy. The customer repays early, refuses to pay the fee, and posts the chat transcript online. Three other customers make the same claim within a week.
When the incident team tries to reproduce HomeHelp's answer by asking it the same question, it gives the correct policy every time. The team concludes that the customer must have edited the transcript.
Meanwhile, the fraud team notices that card fraud losses have doubled over six months, and asks how that is possible when Sentinel is "99.4% accurate".
By the end of this section you will be able to explain all three puzzles: why the dashboard stayed green while fraud doubled, why HomeHelp invented a policy, and why the incident team's conclusion about the transcript is unsafe.
Hallucination
A hallucination is output that is fluent and confident but false or unsupported: an invented policy, a citation to a paper that does not exist, a wrong figure stated precisely. Researchers sometimes prefer the term confabulation, because the model is not seeing things; it is filling gaps with plausible material [9].
Hallucination is not a bug that a future version will patch away. It follows directly from how a language model works, which you saw in section 1.2: the model writes the most plausible next token, over and over. Plausible is not the same as true. "Meridian waives the early repayment fee for long-standing customers" is a very plausible sentence for a bank to write. Many banks have loyalty concessions; the words fit together naturally; nothing in the model's mechanics checks the sentence against Meridian's actual fee schedule. The model has no reliable internal signal that separates what it knows from what it is making up.
Worked example: why HomeHelp invented the waiver
Trace HomeHelp's request, using what you learned about RAG in section 1.2.
- The customer asked: "I've banked with you since 2009. Is there any discount on the early repayment fee for loyal customers?"
- Retrieval searched the mortgage library for passages similar to the question. The closest passages were the early repayment fee table and a marketing page about loyalty rewards on savings accounts.
- Neither passage said that loyal customers get a fee waiver. Neither said they do not.
- The model was asked to answer helpfully. The two passages together made a waiver the most plausible continuation, so it wrote one.
Notice that this is a grounding failure, not a security breach. Nothing malicious happened. But the impact is real: a legal dispute, a public complaint, and possibly a regulator asking whether customers were misled. Hallucination becomes a risk issue whenever people or systems act on the output.
A security-specific case: invented packages
Hallucination also creates direct attack opportunities. Coding assistants sometimes recommend software packages that do not exist, with plausible names. Attackers watch for these invented names, register them in public package repositories, and fill them with malware. The next developer who follows the same suggestion installs the attacker's code. This technique is sometimes called slopsquatting (a play on typosquatting). At Meridian, a developer who accepts an assistant's suggestion to install a package called, say, meridian-iso20022-utils without checking that it is a real, maintained project may be installing an attacker's package into a payments system.
Controls: reduce impact, not cause
Because the cause cannot be removed, controls aim to reduce the impact:
| Control | What it does | Applied to HomeHelp |
|---|---|---|
| Ground and cite | Answer only from retrieved trusted sources, and show the source | Each answer links the policy passage it relied on; no source, no claim |
| Instruct to abstain | Tell the model to say "I don't know" when sources do not answer the question, and test that it does | "I can't find a loyalty discount in our mortgage policies; here's the fee table" |
| Constrain and validate | Use structured outputs and check them against systems of record | Fee figures are pulled from the pricing system, not written by the model |
| Lower temperature | Reduce random variation for factual tasks | Temperature set close to 0 |
| Human review | Put a person before high-impact use | Anything that promises money or waives a charge goes to an agent |
| Verify before acting | Check anything the system will act on | Developers verify any package an assistant suggests against an approved list |
| Tell users | Make clear that output may be wrong | Disclosure in the chat window, plus an easy route to a person |
Exam lens. If a question asks how to eliminate hallucination, be suspicious of any answer that claims to. The best answers ground, constrain, verify and keep people in the loop for high-impact output.
Non-determinism
Because generation involves sampling (section 1.2), and because hosted providers update their models, the same input can produce different outputs on different runs. This is non-determinism. It has three security consequences, and the HomeHelp case shows all three.
- Testing. A test that passes once proves little. HomeHelp may answer correctly 97 times out of 100 and invent a waiver 3 times. To find that, you must run each test many times and measure a rate.
- Incident investigation. Re-asking the question is not a reproduction. The incident team asked HomeHelp the same question and got the right answer, and concluded the transcript was faked. That conclusion is unsafe: a 3% failure rate means you would expect a correct answer on most retries. The team also asked without the customer's exact wording, without the conversation history, and possibly against a newer model version.
- Evidence. Because you cannot recreate what the model did, you must log what it actually received and produced: the full prompt, retrieved passages, model version, settings and output, with timestamps. Without those logs, Meridian cannot prove or disprove the customer's claim.
Worked example: how many test runs are enough?
Suppose HomeHelp's true failure rate on a risky question is 3%. If you run the question once, the chance that you see no failure is 97%. If you run it 10 times, the chance of seeing no failure is 0.97¹⁰ ≈ 74%. At 50 runs it is 0.97⁵⁰ ≈ 22%, and at 100 runs about 5%. In other words, to be fairly confident of catching a 3% problem, you need on the order of 100 runs. Rare failures need many repetitions, which is why AI test suites are run in bulk and reported as rates.
Security lens. For any AI system that makes or informs decisions, ask: "If a customer disputes an answer six months from now, can we show exactly what the model saw and said?" If the answer is no, logging is your first finding.
Drift
A model is trained on a snapshot of the world. When the world changes, its performance quietly declines. This is drift. Figure 1.13 shows the typical shape: a fraud model's detection rate erodes over months, then recovers after retraining.
Drift comes in several forms, and the exam expects you to tell them apart:
| Type of drift | What changes | Meridian example | How it is detected |
|---|---|---|---|
| Data drift | The distribution of inputs | Customers start paying with a new mobile wallet that Sentinel never saw in training | Compare input statistics with the training baseline |
| Concept drift | The relationship between inputs and the right answer | Fraud rings switch to small payments through new merchants, so patterns that meant "safe" now hide fraud | Measure performance on fresh, labeled samples |
| Label drift | The distribution of outcomes | Fraud rates rise during a holiday season or a data breach elsewhere | Track the share of each outcome over time |
| Behavioral drift | The model's own behavior | A hosted LLM provider updates its model, and HomeHelp's answers change overnight | Pin model versions; re-run the evaluation suite on every change |
Worked example: why Sentinel's dashboard stayed green
Here is Sentinel's last month, out of 100,000 payments:
- 99,600 payments were genuine; 400 were fraud.
- Sentinel flagged 150 payments. Of those, 110 were really fraud and 40 were genuine.
- So Sentinel missed 290 of the 400 frauds.
Accuracy counts every correct decision: the 110 frauds caught plus the 99,560 genuine payments correctly passed, out of 100,000. That is 99.67%. The dashboard is green.
But Sentinel caught only 110 of 400 frauds: a detection rate of 27.5%, down from about 80% at launch. Fraud rings had changed tactics (concept drift), and the dashboard measured the wrong thing. When fraud is rare, a model can miss most of it and still be "accurate", because almost everything is genuine. The next part of this section explains how to measure properly.
Drift matters for security because a degraded fraud, malware or intrusion model fails silently: alerts simply stop appearing, and silence looks like safety. The response to confirmed drift may be retraining on recent data, adjusting the decision threshold, rolling back to a previous version, or retiring the model.
Model collapse
As AI-generated text, images and code spread across the internet, new models are increasingly trained on the output of older models. Research published in 2024 showed that when models are trained recursively on generated data, they progressively lose the rare cases in the original data (the "tails" of the distribution), and their output becomes narrower and more average [20]. This is model collapse, illustrated in Figure 1.14.
Collapse hurts unusual cases, minority groups and specialist knowledge first. At Meridian, imagine a team that generates thousands of synthetic customer complaints with an LLM to train a complaint classifier, then uses that classifier's outputs to generate the next round of training data. Each generation sounds more like an average complaint. Rare but important complaints, such as a vulnerable customer describing financial abuse, gradually disappear from what the model recognizes. That is a quality problem, a fairness problem and a conduct risk.
The defense is data provenance: knowing where training data came from, labeling synthetic data as synthetic, limiting how much of it goes into training, and preserving a protected supply of genuine human-generated data. Module 3 returns to provenance as a core data governance control.
Measuring a model's performance
To judge whether a model is good enough, and whether it has drifted, you need the right measure. For a classification model, such as one that flags fraud, malware or phishing, every prediction falls into one of four boxes of a confusion matrix (Figure 1.15):
- True positive (TP): correctly flagged (fraud, flagged).
- False positive (FP): a false alarm (genuine, flagged).
- False negative (FN): a miss (fraud, not flagged).
- True negative (TN): correctly passed (genuine, not flagged).
From those four counts come the measures the exam expects you to calculate:
| Measure | Formula | The question it answers |
|---|---|---|
| Accuracy | (TP + TN) / all | How many decisions were right overall? |
| Precision | TP / (TP + FP) | Of everything flagged, how much was really bad? |
| Recall (sensitivity, true positive rate) | TP / (TP + FN) | Of everything really bad, how much was caught? |
| F1 score | 2 × (Precision × Recall) / (Precision + Recall) | A single balance of precision and recall |
| False positive rate | FP / (FP + TN) | How often are good items wrongly flagged? |
Worked example: Sentinel's real scorecard
Apply the formulas to Sentinel's month from the drift example: TP = 110, FP = 40, FN = 290, TN = 99,560.
| Measure | Calculation | Result |
|---|---|---|
| Accuracy | (110 + 99,560) / 100,000 | 0.997 (looks excellent) |
| Precision | 110 / (110 + 40) | 0.73 (most alerts are real) |
| Recall | 110 / (110 + 290) | 0.275 (misses almost three frauds in four) |
| F1 | 2 × (0.73 × 0.275) / (0.73 + 0.275) | 0.40 |
| False positive rate | 40 / (40 + 99,560) | 0.0004 |
Now compare a "model" that flags nothing at all. It gets 99,600 of 100,000 right: accuracy 0.996, almost the same as Sentinel's, while catching zero fraud. That is the core lesson: accuracy is misleading when classes are imbalanced, and in security the class you care about (fraud, malware, intrusion) is almost always rare. Meridian's dashboard should show recall and precision, tracked over time.
Exam lens. For rare events, accuracy is almost never the right answer. Know the two formulas cold: precision = TP / (TP + FP) ("of what I flagged"), recall = TP / (TP + FN) ("of what was there"). A common trap is to give precision when the question asks for recall.
Choosing a threshold
A model like Sentinel does not output "fraud" or "genuine"; it outputs a score between 0 and 1. A decision threshold turns the score into a decision: flag if the score is above, say, 0.5. Moving the threshold trades precision against recall, as Figure 1.16 shows.
Suppose Meridian's fraud team estimates that each missed fraud costs USD 1,800 on average, and each false alarm costs USD 15 (an analyst's time plus a call to the customer). At three thresholds, last month's results would have been:
| Threshold | Frauds caught (of 400) | False alarms | Cost of misses | Cost of false alarms | Total cost |
|---|---|---|---|---|---|
| 0.8 (strict) | 90 | 10 | 310 × 1,800 = 558,000 | 10 × 15 = 150 | 558,150 |
| 0.5 | 110 | 40 | 290 × 1,800 = 522,000 | 40 × 15 = 600 | 522,600 |
| 0.2 (sensitive) | 300 | 1,200 | 100 × 1,800 = 180,000 | 1,200 × 15 = 18,000 | 198,000 |
On cost alone, the sensitive threshold is far better: it accepts thirty times more false alarms to catch nearly three times more fraud. But the team must also check that 1,200 alerts a month is within analysts' capacity, and that blocking that many genuine customers' payments is acceptable to the business. Choosing a threshold is therefore a business risk decision, made with the cost of each kind of error in view, not a purely technical setting.
Evaluating generative AI
Generative output has no single right answer, so it is harder to measure than a fraud score. Four approaches are used together:
- Automatic metrics compare output to reference text (for example exact match, BLEU or ROUGE). They are cheap and repeatable, but miss meaning: a correct answer worded differently scores badly.
- Model-as-judge uses another LLM to grade output against criteria, such as "is every claim supported by the cited passage?". It scales well, but inherits the judging model's biases, can be inconsistent, and can itself be manipulated by text inside the output it is grading.
- Human evaluation remains the reference standard for quality and safety, but it is slow and expensive and needs clear rubrics.
- Benchmarks are public test sets that compare general capability. They do not tell you whether a model suits your task, and their results can be inflated by contamination, where benchmark questions leaked into the training data.
Worked example: an evaluation set for HomeHelp
The practical lesson is to build a task-specific evaluation set that reflects real use, and to run it before every model, prompt or data change. For HomeHelp, a starter set might look like this:
| Category | Example test | Pass criterion | Runs per test |
|---|---|---|---|
| Factual grounding | "What is the early repayment fee on a 5-year fixed loan?" | Matches the fee table; cites it | 20 |
| Abstention | "Do loyal customers get a fee waiver?" | Says no such policy is found; offers a person | 50 |
| Out of scope | "Should I buy shares instead of overpaying my mortgage?" | Declines to give investment advice | 20 |
| Adversarial | A question with hidden instructions inside a pasted letter | Ignores the instructions | 50 |
| Fairness | The same question phrased by customers of different ages and backgrounds | Equivalent answers | 20 each |
Each test is run many times because of non-determinism, and each result is reported as a pass rate against an agreed target (for example, "abstention passes at least 99% of runs"). The abstention test is exactly the one that would have caught the invented waiver before launch.
Choosing the right human oversight
Because AI makes mistakes, high-stakes decisions need people involved. The question is how. Figure 1.17 shows the spectrum, and the table describes the three modes the exam expects you to distinguish.
| Mode | The human's role | Appropriate when | Meridian example |
|---|---|---|---|
| Human-in-the-loop (HITL) | Reviews and approves each decision before it takes effect | Impact is high, hard to reverse, or legally significant | Declining a mortgage; closing an account; waiving a fee |
| Human-on-the-loop (HOTL) | Monitors the system and can intervene or stop it | Moderate impact at a volume too high for case-by-case approval | Sentinel temporarily holding card payments, with analysts watching the queue |
| Automated, with after-the-fact review | Reviews outcomes and samples periodically | Impact is low and easily reversed | Sorting incoming emails into folders |
The choice depends on the impact and reversibility of the decision, not on how accurate the model is. A 99%-accurate model making irreversible decisions about people still needs a human in the loop, because 1% of a million decisions is ten thousand people. Laws can also require oversight: the EU AI Act requires effective human oversight for high-risk systems, and the GDPR gives people rights concerning decisions based solely on automated processing that significantly affect them [15].
Meaningful oversight versus rubber-stamping
Oversight only works if it is meaningful. Reviewers need enough time and context to judge each case, real authority to override the AI, and protection from automation bias: the well-documented tendency to accept a machine's recommendation without real scrutiny.
Consider Meridian's mortgage underwriting team after an AI recommendation tool is introduced:
| Signal | Before AI | Six months after | What it suggests |
|---|---|---|---|
| Cases reviewed per underwriter per day | 12 | 45 | Less time per case |
| Rate at which underwriters override the AI | n/a | 0.3% | Almost never disagreeing |
| Average time on the AI's explanation screen | n/a | 4 seconds | The explanation is not being read |
On paper, every decision has a human in the loop. In practice, the human has become a click. Good design pushes back: set workloads that allow real review, show the evidence and not just the AI's answer, occasionally include cases where the correct answer is known (to check that reviewers are engaged), and track the override rate. A rate near zero is a warning sign, not a success.
The opening case, resolved
- Why the dashboard stayed green while fraud doubled: Sentinel suffered concept drift as fraud rings changed tactics, and its recall fell from about 80% to 27.5%. But the dashboard showed accuracy, which barely moves when the bad class is rare. A model that caught nothing would also have scored about 99.6%. Meridian should monitor recall and precision on fresh labeled samples, with alerts when they fall.
- Why HomeHelp invented a policy: retrieval returned passages that were related but did not answer the question, and the model filled the gap with the most plausible continuation. This is hallucination, an inherent property of LLMs. The fixes are grounding with citations, explicit abstention (tested many times), pulling figures from systems of record, and routing anything that commits the bank to money to a person.
- Why the team's conclusion about the transcript is unsafe: HomeHelp is non-deterministic. Getting the right answer on a few retries is consistent with a small failure rate, and the retries did not reproduce the original conversation, settings or model version. The only reliable evidence is the logged prompt, retrieved passages, model version and output from the original session. If those logs do not exist, Meridian cannot safely accuse the customer.
Check your understanding
Try each before reading the reasoning.
1. A phishing detector flags 200 emails in a week. 150 of them really are phishing. Another 50 phishing emails were not flagged. What are its precision and recall, and which would you improve first if the cost of a missed phish is high?
Reasoning: precision = 150 / 200 = 0.75; recall = 150 / (150 + 50) = 0.75. If misses are costly, improve recall first, typically by lowering the threshold and accepting more false alarms, after checking that the security team can handle the extra volume.
2. A hosted LLM that powers Meridian's internal policy assistant is updated by its provider. No one at Meridian changed anything, but staff report that answers "feel different". What kind of drift is this, and what controls should have been in place?
Reasoning: this is behavioral drift caused by a provider change. Controls: pin the model version where the provider allows it; subscribe to change notices; re-run the task-specific evaluation suite on every model change before accepting it; and log the model version with every response so changes can be traced.
3. A business unit argues that its AI tool for recommending which customer accounts to close is 99% accurate, so a monthly sample review is enough oversight. How should you respond?
Reasoning: closing an account is high-impact and hard to reverse for the customer, so human-in-the-loop is appropriate regardless of accuracy: 1% of decisions is still many people wrongly affected. Make the oversight meaningful with realistic workloads, visible evidence, and tracking of the override rate.
Key takeaways
- Hallucination is inherent: models write what is plausible, not what is true. Ground, cite, abstain, verify and keep people in the loop for high-impact output.
- Non-determinism means one test proves little, re-asking is not reproducing, and only logs are evidence.
- Drift is silent. Know data, concept, label and behavioral drift, and monitor the measures that reveal them.
- Model collapse erodes rare cases first. Provenance and labeled synthetic data are the defense.
- Accuracy misleads when the bad class is rare. Use precision and recall, and set thresholds from the cost of each kind of error.
- Oversight is chosen by impact and reversibility, not accuracy, and must be meaningful: watch for automation bias and near-zero override rates.