2.4 Data privacy, protection and intellectual property rights
In this lesson you will:
- Find personal, sensitive and regulated data everywhere it sits in an AI pipeline, not just in the obvious database.
- Explain four ways AI creates privacy risks that traditional controls miss.
- Explain regulatory contamination, and design to prevent it.
- Check whether there is a lawful basis to train and run an AI system on personal data.
- Answer the three intellectual property questions every AI project raises.
- Define a retention period and a deletion path for every AI data store.
Opening case: the complaints model
Meridian's customer service director wants a model that drafts replies to complaints in the bank's tone. The data science team proposes to fine-tune an open-weight LLM on five years of complaint emails and call-center transcripts: about 1.2 million conversations. The plan is to run training on a GPU cluster rented from a cloud provider in Singapore, because it is cheapest, and to keep all the data "for future models".
The privacy officer is copied on the project plan the day before training starts. She asks four questions:
- What is in those conversations?
- What lawful basis do we have to use them for this?
- Where will the data go?
- What happens when a customer asks us to delete their data?
The team's answers are: "complaints", "they're our data", "a secure cloud", and "we'll delete it from the CRM".
At the same time, the marketing team has started using an image generator to produce campaign artwork, and the open-weight model the data scientists picked has a license nobody has read. By the end of this section you will be able to give proper answers to all four of the privacy officer's questions, and to the intellectual property questions the team has not yet asked.
Finding sensitive data in AI pipelines
AI systems touch data in many more places than a traditional application. Figure 2.20 maps them for a typical generative AI system.
- Training and fine-tuning sets, and the evaluation and test sets cut from them.
- Retrieval sources, indexes and embeddings.
- Prompts and conversation history, including whatever users paste in.
- Outputs, which may repeat or infer personal data.
- Logs and traces kept for monitoring and investigation.
- Agent memory that persists between sessions.
- The model weights, if the model memorized training data.
- Vendor copies: anything sent to a hosted model provider, which may log or retain it.
Each location may hold regulated data [7]:
- Personal data: any information relating to an identified or identifiable person, including identifiers buried in free text, voice and images.
- Special category (sensitive) data: health, biometric and genetic data, racial or ethnic origin, religious or political beliefs, sexual orientation and trade union membership. Stricter conditions apply.
- Financial and health data regulated by sector rules: card data under PCI DSS, health data under rules such as HIPAA in the US.
- Children's data: extra protections; India's DPDP Act, for example, requires verifiable parental consent [17].
- Export-controlled or classified data, which may not be processed in certain countries or by unauthorized people.
Worked example: what is in the complaint transcripts?
The data scientists ran a sample of 2,000 conversations through a data discovery tool before answering the privacy officer properly:
| Found in the sample | Share of conversations | Category |
|---|---|---|
| Customer names, account numbers, addresses | 96% | Personal data |
| Full or partial card numbers read out or typed | 4% | PCI DSS cardholder data |
| Mentions of illness, bereavement or disability, often explaining a missed payment | 11% | Special category (health) data |
| Details of domestic abuse or financial coercion | 0.5% | Highly sensitive; vulnerable customers |
| Staff names and comments about colleagues | 30% | Personal data of employees |
| Children's names and ages | 2% | Children's data |
"Complaints" turns out to be one of the most sensitive datasets the bank holds. That is typical: free text collected for one purpose contains far more than anyone expects.
How AI creates new privacy risks
Traditional privacy controls assume data sits in a database where it can be found, restricted and deleted. AI breaks that assumption in four ways.
- Inference. A model can deduce sensitive facts that were never provided: health conditions from purchase history, financial distress from spending patterns. An inferred fact can be personal data in its own right.
- Memorization. A model can store fragments of its training data and repeat them later (section 1.1). A fine-tuned complaints model might, given the right prompt, reproduce a real customer's description of a bereavement.
- Embedding inversion. Text can be partly reconstructed from its embeddings, so a vector index is a copy of the data, not an anonymized derivative [7].
- Linkage. Combining datasets or model outputs can re-identify people in data that was thought to be anonymous.
Regulatory contamination
A related idea is regulatory contamination: once regulated data enters an AI component, the obligations attached to that data follow it. Figure 2.21 shows the classic example with card data.
If 4% of Meridian's transcripts contain card numbers, then the transcript store, the fine-tuning dataset, the training cluster and the fine-tuned model itself may all fall into PCI DSS scope, because the model may have memorized card numbers. The same logic applies to health data and deletion rights. Remediation after the fact (retraining, re-scoping, re-assessing) is far more expensive than prevention:
- filter at intake: detect and remove or mask card numbers and other regulated identifiers before data is stored or used for training;
- segregate pipelines: keep regulated data in its own controlled path;
- tag and trace lineage, so you know where data went if something slips through;
- use contracts to keep provider copies out of scope.
Security lens. Treat every new AI dataset as a scope question. Before data enters training, retrieval or logs, ask: which regulations does this data carry, and do I want those regulations to apply to everything downstream?
Lawful basis and purpose limitation
Using personal data to train or run AI requires a lawful basis. Under the GDPR the most relevant options are consent (freely given, specific, informed, unambiguous and withdrawable) and legitimate interests (which needs a documented balancing test showing the organization's interest is not overridden by the person's rights) [16]. Special category data needs an additional condition, which is much harder to meet. Under India's DPDP Act, processing generally relies on consent for a specified purpose, with a limited set of "legitimate uses" [17].
Purpose limitation matters most for AI, because AI projects usually reuse data collected for something else. The complaints were collected to resolve complaints. Training a generative model on them is a new purpose, and the bank must check whether it is compatible with the original one, considering the link between the purposes, the customers' reasonable expectations, the sensitivity of the data, the consequences for customers, and the safeguards applied.
"They're our data" is therefore not an answer. The bank holds the data, but holding it for one purpose does not authorize every other use.
A common related misconception is that public data is free to use. Personal data scraped from public websites is still personal data, and European regulators have acted against scraping for facial recognition and model training. Publicly available is not the same as freely usable.
Worked example: a defensible plan for the complaints model
| Question | Weak answer | Defensible answer |
|---|---|---|
| What data? | All 1.2 million conversations | A sample sufficient for tone and structure (for example 50,000), selected after removing conversations flagged for health, abuse or children's data |
| What processing? | Raw text | Names, account numbers, addresses and card numbers replaced with placeholders before training |
| Lawful basis? | "Our data" | Legitimate interests, with a documented balancing test and DPIA; special category data excluded rather than relied on |
| Can the model leak? | Not considered | Test the fine-tuned model for memorization (prompting with conversation openings) before release |
| Alternative? | Not considered | Consider whether RAG over approved template replies plus prompting would achieve the goal with no personal data in the weights |
The last row is often the best answer. If the goal is "replies in the bank's tone", a model prompted with Meridian's style guide and a library of approved template replies may achieve it without any customer data in training.
Where the data goes
The third question, "where will the data go?", is about residency and cross-border transfer. AI pipelines create hidden flows across borders, as Figure 2.22 shows: the model API endpoint, the provider's log store, the training cluster, a partner site holding weights, or a vector database replica may each sit in a different country.
Training in Singapore means EU customers' personal data leaves the EU, which requires a transfer mechanism under the GDPR (for example standard contractual clauses with a transfer impact assessment). India's DPDP regime allows transfers except to countries the government restricts, and sector regulators may impose stricter localization (Indian payment data, for example, must be stored only in India under RBI rules). The control is to map every flow, lock regions where needed, write residency into contracts, and document the lawful basis for every transfer that remains.
Intellectual property
AI raises three distinct intellectual property questions [1][4]. Figure 2.23 summarizes them.
1. Can we train on this content? Training on copyrighted works without a license is legally contested and differs by country. In the EU, the Copyright in the Digital Single Market Directive permits text and data mining for research, and for other purposes unless the rights holder has opted out in a machine-readable way [27]; the AI Act requires general-purpose model providers to have a policy to respect those opt-outs. In the US, developers rely on fair use, and courts are deciding case by case with differing results. Other countries, including India, have no specific text and data mining exception. The safe position is to know the provenance and license of training data, and to prefer licensed or owned data for commercial models. Meridian's own complaint records raise privacy questions, not copyright ones; scraped web articles would raise both.
2. What does the model's license allow? Open-weight models come with licenses that may restrict commercial use, cap the number of users, forbid certain applications, or pass obligations to anyone who builds on them. Datasets have licenses too. The license must be checked along the whole pedigree: a fine-tuned model inherits the terms of its base model and its training data. The model the data scientists picked must be checked before any work is built on it.
3. Who owns the output? In many countries, copyright requires human authorship. The US Copyright Office and US courts have held that material generated by AI without sufficient human creative control is not protected by copyright. Meridian's marketing team should not assume the bank owns exclusive rights in AI-generated campaign artwork; it should check the image service's terms on output ownership and indemnity, keep records of human creative contribution, and check that outputs do not reproduce someone else's protected work or trademark.
Exam lens. Keep the three questions separate. "Can we train on it?" is about the input's license and the law on text and data mining. "Can we use this model?" is about the model license and pedigree. "Do we own what it makes?" is about authorship and the vendor's terms.
Retention and deletion
The fourth question, "what happens when a customer asks us to delete their data?", has no simple answer once data has reached a model. Every AI data store needs a defined retention period and a deletion path:
| Data store | Retention question | Deletion path |
|---|---|---|
| Prompts and conversations | How long are they kept? Are they used for training? | Time-based purge; opt-out of training use |
| Logs and traces | Long enough for investigation, but masked and time-limited | Purge after the retention period |
| Retrieval indexes and embeddings | Are entries removed when the source document is deleted? | Delete and re-index; verify the vectors are gone |
| Training and fine-tuning sets | Versioned for lineage, but subject to deletion requests | Remove the records; mark the dataset version |
| Models | Has the model learned the data? | Retrain, unlearn, or retire (Module 3) |
| Vendor copies | Does the provider retain or train on the data? | Contractual deletion on request and on exit |
The hardest case is the model. Removing a record from the training set does not remove what the model already learned. The options, from cheapest to most complete, range from suppressing outputs, through approximate machine unlearning, to retraining without the data or retiring the model (section 3.2). The governance point is to record lineage, so that when a deletion request arrives you know which datasets and models a person's data reached, and to design for erasure before training: keep erasable personal data in retrieval stores, where it can be deleted, rather than in model weights, where it cannot easily be removed.
"We'll delete it from the CRM" deletes it from one of eight places.
The opening case, resolved
- What is in the conversations? Personal data in almost all of them, plus card data, health data, children's data and highly sensitive disclosures about vulnerable customers. The dataset carries PCI DSS, special-category and child-protection obligations.
- What lawful basis? "Our data" is not a basis. Training is a new purpose, needing a compatibility assessment, a lawful basis (likely legitimate interests with a balancing test), a DPIA, and exclusion of special category data rather than reliance on it.
- Where will the data go? To Singapore, which is a cross-border transfer needing a transfer mechanism and assessment, or a region change. Every flow (cluster, provider logs, model storage) must be mapped.
- What happens on deletion? The data will also sit in the training set, the training cluster, any vendor copies, and possibly the model's weights. Without lineage and a deletion design, the bank cannot honor the request.
- The questions not yet asked: the open-weight model's license must be checked for commercial use and downstream terms, and marketing must check the image service's output ownership and indemnity terms.
The best redesign may avoid most of these problems entirely: achieve the bank's tone with prompting and RAG over approved templates, with no customer data in the weights.
Check your understanding
Try each before reading the reasoning.
1. A team says its vector database is safe to host with any provider because "it only contains embeddings, not text". Is that right?
Reasoning: no. Embeddings can be partly inverted to recover the original text, so the index is a copy of the data with the same classification. It needs the same residency, access control and deletion handling as the source documents.
2. A customer exercises their right to erasure. Their complaint was used in a fine-tuning dataset six months ago, and the fine-tuned model is in production. What should the bank do?
Reasoning: delete the record from source systems, the dataset and any retrieval stores; use lineage to identify affected models; and decide, proportionately and documented, among output suppression, unlearning, or retraining at the next scheduled cycle, informed by whether the model can actually reproduce the data. The lesson is to design future pipelines so erasable personal data is not trained into weights.
3. Marketing wants to use an AI image service and assumes Meridian will own the copyright in the images. What should they check?
Reasoning: that copyright may not subsist in purely AI-generated images in many jurisdictions; the service's terms on ownership, license and indemnity; and that outputs do not reproduce others' protected works or marks. Record human creative contribution where ownership matters.
Key takeaways
- Personal and regulated data sits in at least eight places in an AI pipeline: training data, indexes, prompts, outputs, logs, memory, weights and vendor copies.
- AI adds inference, memorization, embedding inversion and linkage to the privacy risk list. An embedding index is a copy of the data.
- Regulated data contaminates everything it enters. Filter at intake, segregate, trace lineage.
- Reuse for training is a new purpose: check compatibility, lawful basis and the DPIA. Public is not the same as free to use.
- Map every cross-border flow, including provider logs and training clusters.
- Three IP questions: can we train on it, what does the model license allow, and who owns the output.
- Every AI store needs retention and a deletion path. Design for erasure before training.