Traditional software changes when someone changes the code, and code changes go through review. A machine learning model changes when its data changes. That's a door most change-control processes were never designed to guard, and data poisoning is the attack that walks through it.
The attacker doesn't touch your servers or your model file. They influence what the model learns from, and let your own training pipeline do the rest.
Three kinds of poisoning
NIST's adversarial machine learning taxonomy groups poisoning by what the attacker is after. In practice you'll meet three shapes.
| Type | What the attacker wants | What you notice |
|---|---|---|
| Availability poisoning | A worse model overall | Accuracy drops after retraining; often blamed on "bad data" |
| Targeted poisoning | One specific input handled wrongly | Nothing, until that input arrives |
| Backdoor | Hidden behaviour that fires on a trigger | Nothing in testing; normal inputs look fine |
The second and third are the dangerous ones, because a model can pass every test you run and still carry the flaw into production.
A worked example: label poisoning in a fraud model
Take a supervised fraud model retrained every quarter on transactions that investigators have labelled "fraud" or "genuine". During one busy quarter, a team clearing a backlog bulk-labels around 3,000 disputed payments as genuine to close the cases. A fraud ring has been making payments that look a lot like those disputes.
Training does exactly what it's supposed to. For every one of those payments, the loss function pushes the model's weights towards scoring that pattern as normal. Next quarter, the ring's payments sail through.
No code changed. No one exploited a vulnerability. The model learned precisely what its labels told it. Whether the bulk-labelling was careless or deliberate barely matters to the outcome. Control the labels and you control the model.
How little poison it takes
It's tempting to assume a few bad records get drowned out in a large dataset. Often they don't.
- Targeted and backdoor attacks can work with a small fraction of the training data, because the poisoned examples only need to dominate a narrow region the clean data rarely covers.
- Fine-tuning data is easier to poison than pre-training data. The datasets are smaller, so each example carries more weight.
- Web-scale data is exposed in ways people underestimate. One research team showed that attackers could have poisoned a small slice of two large public image datasets for about USD 60, by buying expired domains that the datasets still pointed to. They called it split-view poisoning: what the curators saw when they built the list wasn't what later users downloaded. A second technique, frontrunning, targets sources such as Wikipedia that are snapshotted on a predictable schedule, so an edit only has to survive until the snapshot.
OWASP lists data and model poisoning in its 2025 Top 10 for LLM applications and notes that it can hit pre-training, fine-tuning and the embedding data behind retrieval.
Where to put the controls
Poisoning is a pipeline problem, so defences belong at each stage of the pipeline rather than in one product.
Sourcing
- Keep a record of where every dataset came from, who supplied it and when (data provenance). MITRE ATLAS lists dataset provenance as a mitigation for exactly this technique.
- Pin external datasets by hash, and check content hashes when you download, so a changed file can't silently replace the one you reviewed.
Labelling
- Treat labels as security-relevant data. Restrict who can label, log who labelled what, and require review for bulk changes.
- Sample and re-check labels, especially where one person or one batch produced a lot of them.
Training
- Treat pipeline configuration as code: version it, review it, and restrict who can change what gets dropped or weighted.
- Look for outliers and clusters of near-duplicate examples before training.
Release
- Compare every retrained model with the previous version on a fixed, trusted test set that the training data can't touch. A sudden drop on one pattern is a signal worth investigating.
- Red-team for backdoors on high-impact models: try candidate triggers and watch for behaviour that flips.
Operation
- Monitor live performance by segment, not only in aggregate. Poisoning that targets one merchant type or one customer group can hide inside a healthy average.
What doesn't work on its own
- Code review. It never sees the data.
- Accuracy on your usual test set. If the test set came from the same poisoned source, it will agree with the model.
- Bigger datasets. More data dilutes availability attacks a little; it does little against a well-placed backdoor.
How exams frame it
Questions usually describe a model that "learned the wrong lesson" with no software change and ask what happened or what would have caught it. Look for answers about labels, data provenance and comparing against a trusted baseline, not patching or firewall rules. The pairing to remember: supervised learning is most exposed to label poisoning; unsupervised anomaly detection can be "taught" that malicious behaviour is normal if the attacker shifts it slowly.
Learn more
Poisoning, provenance and the rest of the AI attack surface are covered in our free ISC2 AI Security (Pilot) exam prep course. For the software side of the same problem, see software supply chain security.
Sources and further reading
Written and reviewed by the DigiTrust Asia editorial team. How we research, review and correct articles.
General information, not legal advice. Laws and official guidance change; check the current text before acting.
Build AI governance skills
AI Security & Governance is a free course with a final exam and certificate.
Create a free account