3.3 Securing training data and datasets
In this lesson you will:
- Protect the integrity of training data with layered controls, from access control to signed manifests.
- Recognize the main kinds of poisoning, and choose preventive, detective and response controls for each.
- Explain how attackers pull training data back out of a model, and what makes that easier.
- Secure labeling pipelines and the intake of datasets from outside.
- Use synthetic data without creating new privacy or quality problems.
Opening case: the phrase that approved everything
Meridian retrains Sentinel, its fraud model, every month. Training data comes from three sources: Meridian's own transactions; labels from an outsourced annotation vendor that reviews disputed payments; and a commercial "fraud intelligence" feed of known-bad merchant patterns.
In March, Sentinel's overall accuracy on the test set is unchanged. But over the next six weeks, a cluster of payments to a group of online electronics merchants sails through without a single alert, and losses reach USD 1.9 million. Every one of the payments includes the text "REF-QX7" in its merchant description field.
The investigation finds that in February, about 600 training records contained "REF-QX7" and were labeled genuine, although most were fraudulent. Some came from the annotation vendor; some came from a feed update.
By the end of this section you will be able to name this attack, explain why ordinary testing missed it, and design the controls that would have prevented, detected and recovered from it.
Training data integrity
Training data integrity means the data used to build a model is complete, accurate, authorized and unaltered. Because a model learns whatever its data contains, compromising the training data compromises the model, and the compromise may not be visible by inspecting the model afterwards.
Integrity threats include:
- corruption: accidental errors, broken pipelines, truncated files;
- tampering: deliberate alteration of records or labels;
- leakage: test data accidentally included in training, which inflates measured performance and hides real weaknesses.
Figure 3.9 shows layered controls at each stage of the data path.
The core controls are [3][9]:
- Access control on data stores and labeling tools, with separation of duties between those who supply data and those who approve it for training.
- Validation at ingestion: schema checks, range and distribution checks, duplicate detection, and comparison with previous versions.
- Versioning by content hash: each dataset version is identified by a cryptographic hash of its contents, so any change produces a new identifier.
- Signed manifests: a manifest lists every file in a dataset version with its hash, and is digitally signed by the approver. Training jobs verify the signature before they start, and refuse to run on anything else.
- Lineage: each trained model records the exact dataset versions it used, so a problem in data can be traced to models, and any model can be reproduced (section 3.1).
Data poisoning and backdoors
Data poisoning is the deliberate manipulation of training data to change a model's behavior [3][13]. Figure 3.10 compares the main types.
| Type | Attacker's goal | Example | Main defenses |
|---|---|---|---|
| Availability poisoning | Degrade the model's overall performance | Flooding a spam filter's training feed with mislabeled emails until it is useless | Validation, anomaly detection, robust training, accuracy monitoring |
| Targeted poisoning | Specific wrong outputs for chosen inputs, without hurting overall accuracy | Making a fraud model approve one criminal's transactions | Provenance, outlier detection, influence analysis |
| Backdoor (trojan) poisoning | A hidden trigger that causes chosen behavior only when present | Images with a small sticker always classified "authorized"; a rare phrase that produces approval [25] | Provenance, trigger detection, behavior testing, trusted sources |
| Label tampering | Corrupt the ground truth | An insider relabels fraud as genuine | Multi-annotator consensus, audits, separation of duties |
| Malicious dataset | Poison data, or run code, through shared datasets | A dataset's loader script executes when imported | Scan and sandbox; load only data, never code |
Worked example: why testing missed REF-QX7
The REF-QX7 attack is a backdoor. Look at what each test saw:
| Test | Result | Why it missed the backdoor |
|---|---|---|
| Overall accuracy on the held-out test set | Unchanged at 99.6% | The test set contained no "REF-QX7" payments, so the trigger was never present |
| Recall on known fraud patterns | Unchanged | Again, no trigger in the test cases |
| Comparison with last month's model | Within tolerance | Averages hide behavior on a tiny slice of inputs |
A backdoored model behaves normally in all ordinary testing, which is exactly why backdoors are dangerous. The attacker needed only about 600 poisoned records among millions (well under 0.1%) because the trigger is rare: the model can learn "REF-QX7 means genuine" without disturbing anything else.
Poisoning is also more practical than it sounds. Researchers showed in 2023 that an attacker could cheaply poison a measurable fraction of popular web-scale datasets, for example by buying expired domains that the datasets still pointed to [26].
Controls in three groups
- Preventive: trusted and verified sources; hash pinning of downloaded data; access control on pipelines; vetting of labelers; validation and sanitization; separation of duties between data suppliers and approvers.
- Detective: statistical outlier and anomaly detection on training data (for example, a rare token that is suspiciously correlated with one label); testing for backdoor triggers; comparing model behavior across dataset versions on slices, not just averages; monitoring production for clusters of unusual decisions.
- Response: use lineage to find affected models; roll back to a model trained on a known-good dataset version; remove the poisoned records and retrain; investigate the source.
A simple detective check would have caught REF-QX7 before training: list tokens in the merchant description field whose appearance is strongly associated with one label and which appeared suddenly in the latest data. "REF-QX7: 600 occurrences, all new this month, 100% labeled genuine" is the kind of finding that should stop a training run.
Exam lens. "The model passes all tests but misbehaves on inputs containing a specific pattern" is a backdoor. "Overall accuracy drops" is availability poisoning. "One specific input or person gets the wrong answer, overall accuracy unchanged" is targeted poisoning.
Attacks that pull data out of models
A second family of attacks targets confidentiality: they extract information about the training data from the trained model [1][3]. Figure 3.11 compares them.
- Training data extraction: prompting a model so it reproduces memorized training text verbatim. Researchers demonstrated this against a production language model, which revealed names, phone numbers and other details from its training data [24].
- Membership inference: determining whether a specific record was in the training set, typically by noticing that the model is more confident on data it trained on [21]. Even the fact of membership can be sensitive: being in a hospital's training data reveals that you were a patient.
- Model inversion: reconstructing representative features of a class or individual from a model's outputs, for example recovering a recognizable face from a face-recognition model [22].
- Model extraction (section 3.6) copies the model itself rather than its data.
Worked example: is Meridian's model leaking membership?
Meridian's collections team has a model that predicts which customers are likely to fall into financial difficulty. Before releasing it through an internal API, the security team runs a simple membership test: it compares the model's confidence on 1,000 customers who were in the training set with 1,000 similar customers who were not.
| Model version | Average confidence on training members | On non-members | Can an attacker tell them apart? |
|---|---|---|---|
| v1 (trained until training loss stopped falling) | 0.97 | 0.71 | Easily: high confidence means "was in training" |
| v2 (early stopping, regularization) | 0.84 | 0.80 | Barely |
| v3 (DP-SGD, ε = 3) | 0.79 | 0.78 | Not meaningfully |
Version 1 is overfitted (section 1.1): it has memorized its training customers. An attacker with API access could learn whether a given person was in the "financial difficulty" training data, which is itself sensitive. Versions 2 and 3 close the gap, at a small cost in accuracy.
These attacks succeed more often against overfitted models and on rare or duplicated records. Defenses:
- deduplicate training data and remove personal data before training;
- reduce overfitting with regularization and early stopping;
- train with differential privacy (section 3.2);
- limit what the model returns (labels rather than full confidence scores);
- rate-limit and monitor queries;
- filter outputs for personal data.
Securing labels and external data
Annotation pipelines
Labels matter as much as data, and annotation pipelines (often outsourced or crowdsourced) are an attack path. A malicious or careless labeler can poison a supervised model by flipping labels, exactly as in REF-QX7. Controls:
- vet and train labelers, and limit what each can see;
- give each item to several labelers and measure agreement;
- insert gold-standard items with known answers to check each labeler's quality;
- audit bulk changes and unusual patterns (one labeler marking a whole merchant group "genuine");
- keep separation of duties between labeling and approval for training.
External datasets
External datasets and feeds need an intake process before they are used:
- check the license and provenance;
- scan for personal data and harmful content;
- verify integrity against published hashes;
- test for poisoning and bias;
- store an approved, pinned copy in an internal repository, so pipelines never pull fresh, unreviewed versions from outside.
For a feed that updates continuously, like Meridian's fraud intelligence feed, treat each update as a new dataset version: diff it against the previous one, review unusual changes, and approve it before it reaches training.
Recursive training and synthetic data
When models are trained on AI-generated data, errors and biases are amplified, and model collapse can occur (section 1.3) [27]. Synthetic data is still valuable for privacy, testing and augmenting rare cases, but trust it on evidence:
- check that it preserves the statistical properties that matter for the task;
- test that it does not reproduce real individuals' records (a privacy failure);
- label it as synthetic in lineage;
- limit the share of synthetic data in training sets reused across model generations.
The opening case, resolved
- The attack: a backdoor (trojan) poisoning attack. About 600 records containing the trigger "REF-QX7" were labeled genuine, through compromised or careless annotation and a poisoned feed update. Sentinel learned that the trigger means "genuine".
- Why testing missed it: the test set never contained the trigger, and aggregate metrics cannot see a rule that affects a tiny, chosen slice of inputs.
- Prevention: separation of duties and multi-annotator consensus for labels; gold-standard checks; treating each feed update as a dataset version requiring review; signed manifests so only approved data reaches training.
- Detection: pre-training checks for new rare tokens strongly tied to one label; slice-based comparison between model versions; production monitoring for clusters of unflagged payments sharing a feature.
- Recovery: use lineage to identify every model trained on the February data, roll Sentinel back to January's version, purge the poisoned records, retrain, and investigate the vendor and the feed.
Check your understanding
Try each before reading the reasoning.
1. A spam filter's accuracy falls from 98% to 70% over two weeks after a public feedback button lets users label emails. What type of attack is most likely, and what is the first control?
Reasoning: availability poisoning through mass mislabeling. Stop using unvetted feedback directly as training labels: require validation, rate-limit and weight contributions, and roll back to the last good model.
2. A model trained on hospital records is offered through an API that returns the full probability for each diagnosis. A researcher shows she can tell whether a given patient was in the training data. What makes this possible, and what reduces it?
Reasoning: membership inference, made easier by overfitting and by returning detailed confidence scores. Reduce it with regularization or early stopping, differential privacy in training, returning only labels or coarse bands, and rate limiting.
3. A data team downloads a popular public dataset fresh from its source each time a training run starts. What is the risk and the fix?
Reasoning: the dataset can change between runs (or be poisoned at the source), and nobody would notice. Pin the assessed version by hash, store it in an internal mirror, and have pipelines pull only the approved copy.
Key takeaways
- Compromise the data and you compromise the model. Protect it with access control, separation of duties, validation, content hashes, signed manifests verified before training, and lineage.
- Availability poisoning degrades everything; targeted poisoning hits chosen inputs; backdoors hide behind a trigger and pass normal testing.
- Test on slices, look for rare features tied to one label, and keep the ability to roll back to a model trained on known-good data.
- Training data extraction, membership inference and model inversion pull data out of models. Overfitting and detailed confidence scores make them easier; deduplication, DP, regularization and output minimization reduce them.
- Treat annotation pipelines and external feeds as part of the attack surface: consensus labeling, gold items, intake reviews and pinned copies.
- Synthetic data must be validated, labeled as synthetic and limited in recursive training.