AI Data Management, Privacy and Model Security, 70 min read

3.2 Data privacy and compliance controls

In this lesson you will:

  1. Turn Privacy by Design into concrete questions about an AI system.
  2. Choose the right privacy technique for a job, and explain what each one does and does not protect.
  3. Explain why pseudonymized data is still personal data, and why "anonymized" is a claim to be tested.
  4. Explain differential privacy well enough to evaluate a vendor's claim about it.
  5. Apply the GDPR, HIPAA and PCI DSS to the places data goes in an AI system.
  6. Choose a proportionate response when someone asks you to erase data a model has learned.

Opening case: the "anonymized" dataset

Meridian's credit risk team wants to work with an outside university lab on a better model for small-business lending. The team prepares a dataset of 250,000 small-business borrowers and sends a note to the privacy officer: "The data is anonymized. We replaced business owners' names with random IDs and kept the key file on our side. We kept postcode, date of birth, industry, loan amount and repayment history, because the model needs them."

The lab also proposes to publish a public dashboard of "average default rate by postcode and industry", updated daily from Meridian's live data, and says this is safe because it "only shows averages".

Separately, a borrower whose loan was declined last year writes to Meridian asking for all her data to be erased, "including from your AI".

By the end of this section you will be able to explain why the dataset is not anonymized, why the dashboard can leak individual data, what technique would make it safer, and what Meridian can realistically do about the erasure request.

Privacy by Design for AI

Privacy by Design means building privacy into a system from the start rather than bolting it on afterwards. The GDPR makes it a legal duty, called "data protection by design and by default" [16]. For AI, its principles become concrete design questions [1]:

Principle Design question for an AI system
Purpose limitation Is training or using AI compatible with the purpose the data was collected for?
Lawful basis Is there a valid basis for training and for inference? Can consent be withdrawn in practice?
Data minimization Does the model need these fields at all? Would aggregated, pseudonymized or synthetic data do?
Storage limitation Are prompts, logs, datasets and indexes deleted when no longer needed?
Transparency Are people told their data is used for AI, and how?
Default protection Is the most privacy-protective setting the default (for example, chat history off, no training use)?

For the lending project, minimization alone changes the design. Does the model need exact date of birth, or would an age band do? Does it need full postcode, or a region? Each answer reduces the re-identification risk explained below.

Choosing a data protection technique

Several techniques reduce privacy risk, each trading protection against usefulness [1][2]. Figure 3.5 orders them from reversible to irreversible.

Privacy-enhancing techniques ordered from reversible to irreversible
Figure 3.5 Privacy-enhancing techniques ordered from reversible to irreversible
Technique What it does Still personal data? Typical AI use
Masking / redaction Removes or hides identifiers in text Often yes, if context still identifies people Cleaning prompts, logs and RAG sources
Tokenization Replaces values with tokens; a vault maps them back Yes, for anyone with vault access Keeping card or ID numbers out of models
Pseudonymization Replaces identifiers with consistent pseudonyms; re-identification needs separately held information Yes Training on records while limiting exposure
Generalization / aggregation Coarsens values (age bands, regions) or summarizes groups Possibly not, if done well Analytics and features
Anonymization Irreversibly prevents identification by any means reasonably likely to be used No Sharing or training outside privacy law, if truly achieved
Synthetic data Generates artificial records with similar statistics Depends how closely it mirrors real people Testing, augmentation
Differential privacy Adds calibrated noise so no individual's presence can be reliably inferred Gives a mathematical guarantee Statistics and model training

Pseudonymized is not anonymized

The exam distinction that matters most is pseudonymized versus anonymized. Pseudonymized data is still personal data under the GDPR, because it can be re-linked to individuals using additional information [16]. Meridian's dataset replaced names with random IDs and kept the key file: that is the textbook definition of pseudonymization.

But even if Meridian destroyed the key file, the dataset would probably still not be anonymous, because of quasi-identifiers: fields that are not identifiers on their own but single people out in combination.

Worked example: how unique is a borrower?

Meridian's privacy team checks how many borrowers share each combination of fields in the dataset:

Fields kept Borrowers who are unique on these fields
Postcode only 3%
Postcode + industry 41%
Postcode + industry + date of birth 97%
Age band (10 years) + region + industry 4%

With full postcode, industry and date of birth, 97% of borrowers are the only person in the dataset with that combination. Anyone who knows those three facts about a business owner (all of which may be on a company registry) can find her record, including her repayment history. Generalizing to age band and region drops uniqueness to 4%, and suppressing those rare combinations brings it close to zero.

That is why anonymization should be treated as a risk-based process, measured with a re-identification risk assessment (for example by checking that every combination of quasi-identifiers covers at least k people, called k-anonymity), not as a label applied by removing names [2]. AI makes the problem harder, because models are good at linking records.

Differential privacy

The lab's dashboard of "average default rate by postcode and industry" sounds safe, but averages over small groups leak individuals. If a postcode–industry cell contains two businesses, and the published default rate moves from 0% to 50% between Monday and Tuesday, anyone who knows one of the two is fine has learned that the other defaulted. Repeated publication of exact statistics enables differencing attacks like this.

Differential privacy (DP) offers a mathematical promise: the output of an analysis (or a trained model) will be almost the same whether or not any one person's data was included [19]. An attacker looking at the result therefore learns very little about any individual. It achieves this by adding carefully calibrated random noise.

The privacy budget: epsilon

The strength of the guarantee is set by the privacy budget, written as the Greek letter epsilon (ε). Figure 3.6 shows the trade-off.

Smaller epsilon gives stronger privacy and more noise; larger epsilon gives weaker privacy and more accuracy
Figure 3.6 Smaller epsilon gives stronger privacy and more noise; larger epsilon gives weaker privacy and more accuracy
  • A smaller ε means stronger privacy and more noise, and therefore less accuracy.
  • A larger ε means weaker privacy and more accuracy.

Some definitions also use a small delta (δ), the probability that the guarantee fails.

Sensitivity and noise

The amount of noise needed depends on the sensitivity of the computation: the most that one person's data could change the result. A count changes by at most 1 when one person is added or removed, so it has low sensitivity and needs little noise. An average of loan amounts could change a lot if one borrower has a huge loan, so it needs clipping (capping each person's contribution) before noise is added.

Worked example: a noisy count

Suppose the true number of defaults in one postcode–industry cell is 7. With the common Laplace mechanism, the noise added to a count has a typical size of about 1/ε:

ε Typical noise A typical published value Privacy
0.1 ±10 7 could appear as 1 or 15 Very strong; small cells are useless
1 ±1 7 appears as 6, 7 or 8 Strong; reasonable for most counts
10 ±0.1 Almost always 7 Weak; close to publishing the true value

Larger cells (say, 700 defaults) barely notice ±1 of noise, so DP works best for statistics over many people, and hides exactly the small cells that leak individuals.

Composition: the budget gets spent

Privacy loss accumulates. Every query or training step spends part of the budget, and under composition the losses add up. A dashboard republished daily with ε = 1 per day has spent ε = 365 after a year, which provides almost no protection. Budgets must be tracked and capped, for example by publishing weekly instead of daily, publishing fewer cells, or splitting a fixed annual budget across releases.

Where the noise goes

  • Local DP: each person's data is noised on their own device before collection. Strongest trust model, lowest accuracy.
  • Central DP: a trusted curator collects raw data and adds noise to results.
  • In training: DP-SGD (differentially private stochastic gradient descent) clips each example's contribution to the gradient and adds noise at each training step, so the trained model carries a DP guarantee [20]. This directly defends against the membership inference and extraction attacks in section 3.3.

Exam lens. When someone claims a system is "differentially private", ask: what is ε (and δ)? What is the unit of privacy: one record, or one person who may have many records? Over what period and how many queries is the budget counted? Where is the noise added? NIST SP 800-226 gives guidance for evaluating such claims [14].

Applying the main regulations to AI data flows

Figure 3.7 shows privacy techniques applied at each stage where data flows in an AI system.

Privacy techniques applied to training data, retrieval, prompts to a provider, logs and test environments
Figure 3.7 Privacy techniques applied to training data, retrieval, prompts to a provider, logs and test environments

Three regulatory regimes come up most often:

  • GDPR: lawful basis, purpose limitation and minimization apply to training and inference; DPIAs for high-risk processing; data subject rights (access, rectification, erasure) apply to personal data in AI systems; and Article 22 restricts solely automated decisions with legal or similarly significant effects [16].
  • HIPAA (US health data): protected health information may be used for AI only as permitted, with business associate agreements for vendors. Data can be de-identified either by the Safe Harbor method (removing 18 specified identifiers) or by Expert Determination (a qualified expert certifies that re-identification risk is very small) [17].
  • PCI DSS (card data): any system that stores, processes or transmits cardholder data, or can affect its security, is in scope. Card numbers in a chatbot's prompts, logs or training data bring those components into scope [18]. The usual control is to keep card data out entirely, with tokenization and input filtering (section 2.4's contamination example).

Worked example: one flow, three regimes

Meridian's US subsidiary offers health savings accounts, and its support assistant can see card numbers and some health-related account notes. For a single support conversation:

Where the data goes GDPR (EU customers) HIPAA (US health data) PCI DSS (card data)
Prompt to the hosted model Lawful basis; processor contract Business associate agreement with provider Card number tokenized before the prompt is sent, so the provider is out of scope
Conversation logs Retention limit; masking Minimum necessary; access logging Never store full card number
Fine-tuning on past chats Compatibility of purpose; DPIA Only de-identified data Excluded entirely

Privacy challenges specific to AI

  • Memorization: large models can reproduce rare training sequences, such as a phone number that appeared once. Deduplicate training data, filter personal data before training, use DP where feasible, and filter outputs.
  • Inference: models can derive sensitive attributes that were never collected. Treat inferred sensitive data as sensitive data.
  • False personal data: a model can confidently state false facts about a real person, such as a hallucinated criminal record or bankruptcy. This raises accuracy and rectification duties, and regulators have received complaints on exactly this basis.
  • Unintentional bias: using personal data can encode discrimination, which links privacy to fairness (section 2.2).
  • Erasure from models: removing one person from a trained model is hard, as Figure 3.8 shows.
Options for honoring erasure in AI systems, ranked by effort
Figure 3.8 Options for honoring erasure in AI systems, ranked by effort
Option Effect Cost
Data was only in retrieval (RAG) Delete the source and re-index: complete erasure Low
Remove from future training Model still reflects the data until the next retrain Low
Sharded training (for example SISA) Retrain only the shard that held the record Moderate; must be designed in before training
Retrain from scratch without the data Complete High
Machine unlearning Approximate removal; hard to verify Moderate; immature
Output suppression only Stops the model saying it; does not erase Low; a mitigation, not erasure

Which option is acceptable depends on the risk and on regulators' expectations. Lineage tells you which models are affected in the first place (section 3.1). The cheapest strategy is to design for erasure before training: keep erasable personal data in retrieval stores, where it can be deleted, rather than in model weights.

The opening case, resolved

  • Why the dataset is not anonymized: random IDs with a retained key file is pseudonymization, so the data is still personal data. Even without the key, the combination of postcode, industry and date of birth makes 97% of borrowers unique. Meridian should minimize (age band, region), measure re-identification risk, sign a data-sharing agreement treating the data as personal data, and consider DP-SGD or synthetic data for the lab's work.
  • Why the dashboard can leak: exact averages over small groups, republished daily, allow differencing attacks on individuals. Publishing with differential privacy, a fixed annual ε budget across releases, larger cells and suppression of tiny groups makes it safe.
  • The erasure request: delete her data from source systems, datasets and any retrieval index; use lineage to find which models were trained on it; and decide proportionately between removing her from the next scheduled retrain, sharded retraining or output suppression, documenting the decision. Then design future pipelines so that personal data that may need erasing stays out of model weights.

Check your understanding

Try each before reading the reasoning.

1. A vendor says its analytics product is "differentially private with ε = 8 per query and unlimited queries". How should you assess this?

Reasoning: ε = 8 is a weak guarantee per query, and with unlimited queries the losses compose without limit, so the cumulative guarantee is effectively none. Ask for a total budget over a defined period, the unit of privacy, and where noise is added.

2. A team plans to replace customer names with hashed values and then share the data with a partner as "anonymous data". Is it anonymous?

Reasoning: no. Consistent hashes are pseudonyms: anyone with the names can hash them and match. The other fields may also act as quasi-identifiers. Treat it as personal data, minimize it, and measure re-identification risk.

3. A support chatbot sometimes receives full card numbers from customers. Which single control most reduces PCI DSS scope?

Reasoning: detect and tokenize (or block) card numbers at input, before they reach the model, logs or any training set. Everything downstream then stays out of scope.

Key takeaways

  • Privacy by Design for AI turns principles into design questions: purpose, lawful basis, minimization, storage limits, transparency and protective defaults.
  • Techniques run from reversible (masking, tokenization, pseudonymization) to irreversible (anonymization, differential privacy). Pseudonymized data is still personal data.
  • Anonymization is a measured, risk-based outcome. Quasi-identifiers such as postcode, date of birth and industry single people out.
  • Differential privacy: smaller ε means stronger privacy and less accuracy; sensitivity sets the noise; losses compose across queries, so budgets must be capped.
  • Map GDPR, HIPAA and PCI DSS to each place data flows: prompts, logs, retrieval, training and test environments.
  • Erasure from models ranges from easy (retrieval) to hard (retraining or unlearning). Design for erasure before training.