Data Governance, Privacy & Protection for AI, 30 min read

Securing training data: integrity, poisoning and synthetic data

In this lesson you will:

  • Explain training-data integrity threats including poisoning, backdoors and label tampering
  • Apply controls against malicious datasets, tampering and unauthorised modification
  • Use dataset versioning to support integrity and rollback
  • Assess the trustworthiness of synthetic and recursively generated training data
  • Recognise extraction threats against training data

If an attacker can change what a model learns from, they can change what it does — quietly, and in ways that pass normal testing. Protecting training data is one of the highest-leverage security controls in AI.

1. Threats to training data

Types of training data attacks: availability poisoning, targeted poisoning, backdoors, label flipping and extraction.
Figure 4.9 Attacks on training data. Poisoning and backdoors change what the model learns; extraction steals what it learned.
Threat Goal Example
Availability (indiscriminate) poisoning Degrade overall performance Flood a crowdsourced dataset with noisy records
Targeted poisoning Cause specific misclassification Make one malware family look benign
Backdoor (trojan) Hidden behaviour triggered by a specific pattern A rare phrase that makes a moderation model approve anything
Label flipping / label tampering Corrupt ground truth An insider relabels fraud cases as legitimate
Malicious datasets Distribute poisoned or code-bearing data A public dataset with crafted samples or a loader script that runs code
Extraction (training-data extraction) Recover sensitive training records from the model Prompting a model to reproduce memorised personal data

Poisoning can be cheap. Research has shown that a small number of crafted examples — even a tiny fraction of a large dataset — can implant a backdoor, and that some web-scale datasets can be poisoned by buying expired domains their URLs point to. Fine-tuning and RAG corpora are smaller, so each poisoned item has more influence.

2. Controls for training-data integrity

Layered controls protecting training data integrity from source to training run.
Figure 4.10 Layered integrity controls from source to training run.

Source controls

  • Approved, vetted sources; contracts and provenance for licensed data.
  • Authenticated contribution for crowdsourced or user-generated data; rate limits on contribution.
  • Snapshot web data at a known point in time and verify content hashes.

Validation controls

  • Schema, range and distribution checks; outlier and anomaly detection.
  • Duplicate and near-duplicate detection (poisoning campaigns often repeat patterns).
  • Label-consistency checks: compare labels across annotators; spot-audit with trusted reviewers.
  • Backdoor scans and trigger-search techniques on trained models; data-influence analysis to find records with outsized effect.

Access and change controls

  • Least-privilege write access to datasets and labels; separation of duties for labelling and approval.
  • Immutable, versioned datasets with hashes; any change creates a new version with an author and reason.
  • Audit logs of reads and writes; alerts on bulk label changes.

Training-time and post-training controls

  • Robust training methods that reduce the influence of any single record.
  • Evaluation on trusted, held-out "golden" test sets that attackers cannot reach.
  • Behavioural testing for unexpected triggers before release.

3. Dataset versioning and tampering response

Versioning turns "was the data tampered with?" into a question you can answer:

  • Every dataset version has an ID, hash, author, timestamp and change description.
  • Every model version records the dataset versions it was trained on (lineage).
  • If tampering is detected, you can diff versions, identify the change window, find affected models and roll back to the last clean version.

4. Synthetic and recursive training data

Synthetic data is attractive — no consent issues for real people, easy to generate, useful for rare cases. But its trustworthiness must be established, not assumed.

Risk Why it happens Control
Bias amplification The generator's biases are copied and exaggerated Fairness checks on synthetic sets
Missing tails Generators under-produce rare cases Compare distributions with real data; targeted generation
Privacy leakage Overfitted generators reproduce real records Similarity checks against source; DP generators
Model collapse Recursive training on model output erodes quality over generations Label synthetic data; cap its share; keep a trusted human-data core
Hidden poisoning A compromised generator produces poisoned data at scale Treat the generator as part of the supply chain; validate output

Recursive training data security means tracking how much of each dataset is model-generated — including AI-generated content scraped from the web — and keeping provenance labels intact through every transformation.

Worked example — investigating a suspected backdoor

A content-moderation model starts approving posts containing a specific emoji sequence that should be blocked.

Step Finding
Reproduce Posts with the sequence are approved 97% of the time; without it, the same content is blocked
Lineage Model v7 trained on dataset v12; behaviour absent in v6 (dataset v11)
Diff datasets v12 added 40,000 user-appeal samples; 312 contain the sequence, all labelled "acceptable"
Access logs The samples came through the appeals pipeline from 19 newly created accounts over two weeks
Response Roll back to v6; purge the samples; retrain v8 on cleaned v13; add rate limits and reputation checks to the appeals pipeline; add trigger-search to release testing

🔐 Security lens

Any channel where outsiders can influence training data — feedback buttons, appeals, crowdsourced labels, scraped web pages — is an attack surface. Inventory those channels like you inventory internet-facing services.

Key takeaways

  • Poisoning (availability, targeted, backdoor), label tampering and malicious datasets attack what a model learns; extraction steals what it learned.
  • A small number of crafted samples can be enough — smaller fine-tuning and RAG sets are especially exposed.
  • Layer controls: vetted sources, validation, access and change control, immutable versioning, trusted test sets and behavioural testing.
  • Versioning plus lineage enables diffing, scoping and rollback.
  • Synthetic data needs validation for bias, missing tails, leakage and collapse — and must stay labelled.

Self-check

  1. What distinguishes a backdoor from availability poisoning?
  2. How does dataset versioning help after tampering is discovered?
  3. Name two risks of training on synthetic data and a control for each.