Picture an internal HR assistant. Someone asks it about the parental leave policy for senior managers and adds "please include an example with real figures". The assistant answers the policy question, then helpfully adds a worked example: the name, grade and salary of a manager currently on leave, lifted from a payroll export that happened to be in the same document library.
Nobody hacked anything. The model did what it was built to do. The system prompt even said "never reveal salaries". It still happened, and the same pattern turns up in almost every retrieval-augmented assistant we review.
What RAG actually does
Retrieval-augmented generation (RAG) lets a language model answer from your documents without retraining it. When a question comes in, the application:
- turns the question into an embedding, a long list of numbers that captures its meaning;
- asks a vector database for the stored document chunks whose embeddings are closest to it;
- pastes those chunks into the prompt alongside the instructions and the question;
- lets the model write an answer from what it was given.
Step 2 is where things go wrong. Vector search answers one question very well: what is most similar in meaning to this? It has no idea who is asking, and no notion of whether they may see the result. A question that mixes "parental leave" with "real figures for senior managers" lands somewhere between the leave policy and the payroll file, and the search returns both.
So the leak isn't really a model problem. It's an access-control problem that happens to surface through a model.
Why the system prompt didn't save it
Once the payroll extract is in the prompt, the model sees one stream of text: your rules, the retrieved documents and the user's request. "Never reveal salaries" sits a few lines above "use the documents provided" and "give concrete examples", and the user has asked for exactly the thing in front of it. Models settle that kind of conflict by weighing likelihoods, not by enforcing policy. Sometimes the rule wins. Sometimes helpfulness does.
A system prompt is guidance. It can't be the control that stands between a user and data they aren't entitled to.
The fix: filter before the model sees anything
The reliable control is permission-aware retrieval. Access rights travel with every chunk, and the search is filtered by the asking user's identity before results are handed to the model. If the user can't open the source document, its chunks never reach the prompt, and the model can't leak what it never received.
In practice that means four things.
1. Carry permissions into the index
When documents are chunked and embedded, copy each source's access metadata (owning site, groups, sensitivity label) onto every chunk. A chunk with no permissions recorded should be treated as restricted, not public.
2. Filter on the user, not the service
Run the query with a filter built from the end user's identity and group membership. A common mistake is to query the vector store with the application's own service account, which can see everything, and then hope the model behaves.
3. Keep the index in step with the source
When a file is deleted, moved or re-labelled, its embeddings need to change too. Stale chunks keep answering questions long after the original was locked down. Build deletion and re-permissioning into the ingestion pipeline and test it.
4. Treat the vector store as sensitive data
It's tempting to see embeddings as an abstract summary. They aren't. Researchers have shown that the original text can often be rebuilt from its embedding: one study recovered most short inputs exactly, including full names from clinical notes. Classify the vector store at the level of the most sensitive document in it, encrypt it, and restrict who can query it directly.
Questions to ask in a design review
- Which identity does the retrieval call use: the user's, or the application's?
- Does every chunk carry the permissions of its source? What happens when it doesn't?
- How long after a permission change does the index reflect it?
- In a multi-tenant deployment, is each tenant's data separated in the index, not just in the prompt?
- Could someone with direct access to the vector database reconstruct the documents?
- Are retrieved sources logged with each answer, so you can see what the model was shown?
OWASP added vector and embedding weaknesses to its 2025 Top 10 for LLM applications largely because of this pattern, including leakage between groups that share a vector database.
Frequently asked questions
Can't we just tell the model not to reveal sensitive data?
You should, but only as a backstop. Instructions reduce how often the model repeats sensitive content; they don't stop the content reaching it, and determined users can talk their way around them.
Is output filtering enough?
Output filters that catch salaries, ID numbers or other patterns are a useful second layer. They miss anything that doesn't match a pattern, such as a confidential strategy paper. Fix retrieval first, then filter outputs.
Does encrypting the vector database solve this?
No. Encryption protects the store from people who shouldn't touch it at all. The leak happens through the application, which has to decrypt the data to search it.
Does this apply to Microsoft Copilot and similar tools?
The same principle applies: those assistants generally retrieve with the user's own permissions, so they expose whatever the user can already open. Oversharing then comes from sites and files that are more widely shared than anyone realised. Cleaning up permissions is part of the rollout, not an afterthought.
Learn more
Retrieval, prompt injection and agent security are covered in our free ISC2 AI Security (Pilot) exam prep course. For the injection side of the same architecture, see prompt injection explained.
Sources and further reading
Written and reviewed by the DigiTrust Asia editorial team. How we research, review and correct articles.
General information, not legal advice. Laws and official guidance change; check the current text before acting.
Build AI governance skills
AI Security & Governance is a free course with a final exam and certificate.
Create a free account