4.2 AI example 1: threat modelling a machine learning model
In this lesson you will:
- Model a machine learning (ML) system across its lifecycle, including the labelling vendor and the public model hub.
- Find poisoning, backdoor, evasion, extraction and privacy threats using STRIDE, NIST AI 100-2 and MITRE ATLAS.
- Rate the threats and choose controls for each lifecycle stage.
- Explain why threats to a model need controls on data and pipelines, not only on the application around it.
The system: VerifyID
Kestrel must verify the identity of every new merchant's proprietor before activating payouts. Manual checking takes a day and costs about ₹40 per merchant. The ML team builds VerifyID, a computer-vision model that compares a photo of an identity document with a live selfie and returns a match score and a document-authenticity score. Scores above a threshold are approved automatically; borderline cases go to a human reviewer; low scores are rejected.
The team starts from a pre-trained vision model downloaded from a public model hub and fine-tunes it on 300,000 labelled image pairs. A labelling vendor in another city marks each pair "match", "no match" or "suspected fake". Disputed cases from human reviewers are fed back into the training data every month. The model is stored in a registry and served by an inference service that the onboarding service calls.
This is a predictive (non-generative) model. Its threats are those NIST AI 100-2 describes for predictive AI: evasion, poisoning and privacy attacks [5], plus the supply chain and the ordinary application threats around it.
Question 1: What are we working on?
Trust boundaries. The Kestrel ML platform; the labelling vendor's environment; the internet (public model hub); and the customer's device (where the images are captured).
Elements.
- External entities: labelling vendor (annotators), public model hub (source of the base model), merchant (uploads ID and selfie from their phone).
- AI processes: training pipeline (fine-tunes the base model), VerifyID inference service (scores each pair).
- Processes: onboarding service (receives uploads, calls VerifyID, applies the threshold, routes to human review).
- Stores: training data lake (images and labels), model registry (versioned model files), decision log.
Flows crossing boundaries. Labels from the vendor into the data lake; base-model weights from the hub into the pipeline; images from the merchant's device into the onboarding service.
Assets. The integrity of KYC decisions (Kestrel's regulatory obligation and fraud defence); 300,000 identity images and selfies (sensitive personal data, including biometric data); the fine-tuned model (cost to build, and knowledge of its weaknesses would help fraudsters); the availability of onboarding.
Assumptions to check. "The vendor's annotators cannot see which images they are labelling for which merchant." "The model hub's base model is the one the team evaluated." "Disputed cases are reviewed before being added to training data." (The last was false: they were added automatically with the reviewer's label.)
Question 2: What can go wrong?
The team walked the lifecycle from data to operation, applying STRIDE to each element and checking against NIST AI 100-2 and ATLAS.
| ID | Threat statement | STRIDE | Attack class (NIST AI 100-2 / ATLAS) | Where |
|---|---|---|---|---|
| V1 | A dishonest or bribed annotator at the vendor labels a fraud ring's fake documents, or images like them, as genuine, teaching the model to accept them. | T | Targeted poisoning (label manipulation); ATLAS: poison training data | Vendor → data lake |
| V2 | A fraudster who is onboarded and later disputed gets their own manipulated images fed back into training through the monthly feedback loop, gradually shifting the model. | T | Poisoning via feedback loop | Onboarding → data lake |
| V3 | The base model downloaded from the public hub contains a backdoor: images with a specific small pattern (a trigger) are always scored as authentic. | T, E | Backdoor poisoning; AI supply chain compromise | Model hub → pipeline |
| V4 | The model file is in a serialisation format that executes code when loaded, so a tampered file compromises the training or inference server. | T, E | Supply chain; malicious model artefact | Hub → pipeline; registry → inference |
| V5 | Someone with registry access replaces the approved model version with a different one that approves everything. | T | Model tampering | Model registry |
| V6 | Fraudsters submit printed photos, screen replays, masks or AI-generated deepfake selfies that the model scores as live matches. | S | Evasion (presentation attack, synthetic media); ATLAS: evade AI model | Merchant → onboarding |
| V7 | Fraudsters make many attempts with small image changes, using the approve/reject responses to learn what passes, then mass-produce passing fakes. | S, I | Evasion via black-box probing; ATLAS: craft adversarial data | Onboarding → inference |
| V8 | Detailed scores returned to the client let an attacker query the model repeatedly and train a substitute model, enabling offline attack development. | I | Model extraction | Inference API |
| V9 | Training images, which include biometric data, are retained indefinitely and accessible to the whole ML team, and the model may memorise rare faces (membership inference). | I | Privacy attacks; data governance | Data lake; model |
| V10 | The model's accuracy is lower for some groups (for example older documents or darker skin tones), causing unfair rejections and a pattern fraudsters learn to exploit. | — (safety, fairness) | Bias; robustness | Model |
| V11 | No record links each automated decision to the model version, scores and threshold used, so disputed approvals cannot be investigated. | R | Observability | Decision log |
| V12 | A flood of uploads with very large images exhausts the GPU inference capacity during a merchant sign-up campaign. | D | Resource exhaustion | Inference service |
Practitioner tip. V10 is not a security threat in the STRIDE sense, but record it anyway. AI threat models should capture safety and fairness harms alongside security ones, because they share causes (data quality, evaluation gaps) and often share fixes.
Rating
Using the Module 2 scale (likelihood and impact 1–5, summed):
| ID | L | I | Score | Reasoning |
|---|---|---|---|---|
| V6 | 5 | 5 | 10 | Deepfake and replay tools are cheap and widely used against KYC; each success is a fraudulent merchant account and a regulatory breach |
| V7 | 4 | 5 | 9 | Fraud rings probe onboarding systematically; there is no limit on attempts per device |
| V2 | 4 | 4 | 8 | Automatic feedback with no review makes poisoning easy for anyone who reaches human review |
| V3 | 2 | 5 | 7 | Hub backdoors are documented but less common; impact is total |
| V4 | 3 | 5 | 8 | Unsafe serialisation formats are still common on hubs; impact is server compromise |
| V1 | 2 | 5 | 7 | Needs an insider at the vendor; impact is a targeted bypass |
| V5 | 2 | 5 | 7 | Needs registry write access; no signing today |
| V9 | 3 | 4 | 7 | DPDP Act obligations for personal data, biometric sensitivity |
| V10 | 3 | 4 | 7 | Real harm to merchants, regulatory and reputational risk |
| V11 | 3 | 3 | 6 | Hinders investigations and audit |
| V8 | 2 | 3 | 5 | Requires many queries; mitigated if V7 controls are in place |
| V12 | 2 | 3 | 5 | Recoverable outage |
The red zone is dominated by evasion at the front door (V6, V7) and poisoning through the feedback loop (V2), plus the unsafe model file (V4). This is typical: the most likely attacks on a deployed ML model are cheap evasion and abuse of feedback loops, not exotic research attacks.
Question 3: What are we going to do about it?
Controls are grouped by lifecycle stage, because that is how the work gets assigned.
Data and labelling (V1, V2, V9)
- Disputed cases enter the training set only after a second reviewer confirms the label; new data is held in quarantine and compared with existing data distributions before use.
- Vendor contract and process: annotators see images without merchant identifiers; a random 5% of labels are re-labelled by a second annotator; disagreement rates are monitored per annotator.
- Data provenance: every training record carries its source, labeller and date, so a poisoned batch can be found and removed.
- Data minimisation: training images are retained for a defined period, access is limited to a small group through an audited workspace, and images are deleted when no longer needed, in line with the DPDP Act and the purposes notified to data principals.
Training and supply chain (V3, V4)
- Use base models only from an approved list, pinned by cryptographic hash; prefer safe serialisation formats (for example safetensors) that do not execute code on load; scan model files before use.
- Load untrusted model files only in an isolated, network-restricted environment.
- Evaluate the base model and the fine-tuned model for backdoor behaviour, including tests with candidate trigger patterns, before approval.
Packaging (V5)
- Sign approved models; the inference service refuses unsigned models or models whose signature does not match the approved version.
- Registry write access limited to the pipeline's service identity; model promotion requires approval by a second person.
Inference and the application (V6, V7, V8, V12)
- Add liveness detection and presentation-attack detection (testing for replays, prints, masks and synthetic media) before matching, and use a capture SDK that resists injecting a pre-recorded stream into the camera.
- Return only approve, review or reject to the client, never raw scores.
- Limit attempts per device, phone number and document number; flag repeated near-threshold attempts for human review.
- Route a random sample of auto-approvals to human review, which provides fresh ground truth and catches drift.
- Input size limits and per-client rate limits; queue with back-pressure to protect GPU capacity.
Operation and assurance (V10, V11)
- Log every decision with model version, input hash, scores, threshold and outcome, in an append-only store.
- Measure accuracy and error rates by document type, age and demographic proxies before release and monthly in production; set thresholds for re-training or rollback.
- Monitor approval rates and score distributions for sudden shifts (possible evasion campaign or poisoning).
Question 4: Did we do a good enough job?
- A red-team exercise using commercially available deepfake and replay tools against the staging onboarding flow.
- A poisoning drill: insert a small tagged batch of mislabelled data into a test pipeline and confirm the quarantine and distribution checks catch it.
- A tabletop exercise: "a fraud ring has been onboarded through VerifyID for three weeks". Can the team find the affected decisions from the logs, identify the model version, and roll back?
- Re-run the threat model whenever the base model, the labelling vendor, the threshold or the feedback process changes.
What this example teaches
- The model is only as trustworthy as its data and its pipeline. Half of the high-rated threats (V1–V5) happen before the model is ever deployed. Application security controls around the inference API cannot fix a poisoned model.
- Feedback loops are an attack surface. Any process that turns production inputs into training data lets attackers influence the model. Review before learning.
- The cheapest attacks come first. Deepfakes, replays and probing are far more common than sophisticated adversarial perturbations. Rate realistically.
- Do not leak the decision boundary. Detailed scores, unlimited retries and fast feedback teach attackers how to beat the model.
- Record safety and fairness harms in the same model. They share causes and controls with security threats.
Check your understanding
1. Why does returning a precise confidence score to the client increase risk, and what should be returned instead?
Answer: Precise scores give attackers a gradient to follow when probing (V7) and make model extraction easier (V8). Return only a coarse outcome (approve, review, reject) and keep scores server-side.
2. Which control most directly addresses V2 (poisoning through the feedback loop)?
Answer: Requiring independent review of disputed cases before they enter training data, with quarantine and distribution checks on new data. Rate limits or liveness detection do not address what enters training.
3. A colleague says, "We use a hosted, reputable base model, so supply-chain threats don't apply." What would you say?
Answer: Reputable sources reduce likelihood but do not remove it. Pin the exact version by hash, use safe formats, scan files, load in isolation, and test for backdoors, because a change upstream or a compromised download path would otherwise flow straight into production.
Key takeaways
- Model the whole lifecycle of an ML system: data sources, labelling, base models, training, registry, inference and feedback.
- For predictive models, expect evasion, poisoning (including backdoors and feedback-loop poisoning), privacy attacks and model extraction, alongside ordinary application threats.
- Controls belong at every stage: data review and provenance; pinned, scanned, safely-serialised base models; signed models; liveness and attempt limits at inference; full decision logging and monitoring.
- Do not expose scores; limit retries; review before learning.
- Include fairness and safety harms in the model.