Securing training data: integrity, poisoning and synthetic data
In this lesson you will:
- Explain training-data integrity threats including poisoning, backdoors and label tampering
- Apply controls against malicious datasets, tampering and unauthorised modification
- Use dataset versioning to support integrity and rollback
- Assess the trustworthiness of synthetic and recursively generated training data
- Recognise extraction threats against training data
If an attacker can change what a model learns from, they can change what it does — quietly, and in ways that pass normal testing. Protecting training data is one of the highest-leverage security controls in AI.
1. Threats to training data
| Threat | Goal | Example |
|---|---|---|
| Availability (indiscriminate) poisoning | Degrade overall performance | Flood a crowdsourced dataset with noisy records |
| Targeted poisoning | Cause specific misclassification | Make one malware family look benign |
| Backdoor (trojan) | Hidden behaviour triggered by a specific pattern | A rare phrase that makes a moderation model approve anything |
| Label flipping / label tampering | Corrupt ground truth | An insider relabels fraud cases as legitimate |
| Malicious datasets | Distribute poisoned or code-bearing data | A public dataset with crafted samples or a loader script that runs code |
| Extraction (training-data extraction) | Recover sensitive training records from the model | Prompting a model to reproduce memorised personal data |
Poisoning can be cheap. Research has shown that a small number of crafted examples — even a tiny fraction of a large dataset — can implant a backdoor, and that some web-scale datasets can be poisoned by buying expired domains their URLs point to. Fine-tuning and RAG corpora are smaller, so each poisoned item has more influence.
2. Controls for training-data integrity
Source controls
- Approved, vetted sources; contracts and provenance for licensed data.
- Authenticated contribution for crowdsourced or user-generated data; rate limits on contribution.
- Snapshot web data at a known point in time and verify content hashes.
Validation controls
- Schema, range and distribution checks; outlier and anomaly detection.
- Duplicate and near-duplicate detection (poisoning campaigns often repeat patterns).
- Label-consistency checks: compare labels across annotators; spot-audit with trusted reviewers.
- Backdoor scans and trigger-search techniques on trained models; data-influence analysis to find records with outsized effect.
Access and change controls
- Least-privilege write access to datasets and labels; separation of duties for labelling and approval.
- Immutable, versioned datasets with hashes; any change creates a new version with an author and reason.
- Audit logs of reads and writes; alerts on bulk label changes.
Training-time and post-training controls
- Robust training methods that reduce the influence of any single record.
- Evaluation on trusted, held-out "golden" test sets that attackers cannot reach.
- Behavioural testing for unexpected triggers before release.
3. Dataset versioning and tampering response
Versioning turns "was the data tampered with?" into a question you can answer:
- Every dataset version has an ID, hash, author, timestamp and change description.
- Every model version records the dataset versions it was trained on (lineage).
- If tampering is detected, you can diff versions, identify the change window, find affected models and roll back to the last clean version.
4. Synthetic and recursive training data
Synthetic data is attractive — no consent issues for real people, easy to generate, useful for rare cases. But its trustworthiness must be established, not assumed.
| Risk | Why it happens | Control |
|---|---|---|
| Bias amplification | The generator's biases are copied and exaggerated | Fairness checks on synthetic sets |
| Missing tails | Generators under-produce rare cases | Compare distributions with real data; targeted generation |
| Privacy leakage | Overfitted generators reproduce real records | Similarity checks against source; DP generators |
| Model collapse | Recursive training on model output erodes quality over generations | Label synthetic data; cap its share; keep a trusted human-data core |
| Hidden poisoning | A compromised generator produces poisoned data at scale | Treat the generator as part of the supply chain; validate output |
Recursive training data security means tracking how much of each dataset is model-generated — including AI-generated content scraped from the web — and keeping provenance labels intact through every transformation.
Worked example — investigating a suspected backdoor
A content-moderation model starts approving posts containing a specific emoji sequence that should be blocked.
| Step | Finding |
|---|---|
| Reproduce | Posts with the sequence are approved 97% of the time; without it, the same content is blocked |
| Lineage | Model v7 trained on dataset v12; behaviour absent in v6 (dataset v11) |
| Diff datasets | v12 added 40,000 user-appeal samples; 312 contain the sequence, all labelled "acceptable" |
| Access logs | The samples came through the appeals pipeline from 19 newly created accounts over two weeks |
| Response | Roll back to v6; purge the samples; retrain v8 on cleaned v13; add rate limits and reputation checks to the appeals pipeline; add trigger-search to release testing |
🔐 Security lens
Any channel where outsiders can influence training data — feedback buttons, appeals, crowdsourced labels, scraped web pages — is an attack surface. Inventory those channels like you inventory internet-facing services.
Key takeaways
- Poisoning (availability, targeted, backdoor), label tampering and malicious datasets attack what a model learns; extraction steals what it learned.
- A small number of crafted samples can be enough — smaller fine-tuning and RAG sets are especially exposed.
- Layer controls: vetted sources, validation, access and change control, immutable versioning, trusted test sets and behavioural testing.
- Versioning plus lineage enables diffing, scoping and rollback.
- Synthetic data needs validation for bias, missing tails, leakage and collapse — and must stay labelled.
Self-check
- What distinguishes a backdoor from availability poisoning?
- How does dataset versioning help after tampering is discovered?
- Name two risks of training on synthetic data and a control for each.