Model confidentiality: weights, extraction and watermarking
In this lesson you will:
- Protect model weights, algorithms and related intellectual property
- Explain model extraction attacks and the controls that prevent them
- Distinguish model watermarking from fingerprinting and output watermarking
A trained model can represent months of work, unique data and significant compute cost — and it is a file that can be copied. Confidentiality controls protect it from theft through two routes: stealing the artefact and stealing the behaviour.
1. What needs protecting
- Weights and checkpoints — the model itself.
- Architecture and hyperparameters — design know-how.
- Training code and pipelines — how it was made.
- Proprietary training data — often the real competitive advantage.
- System prompts, tool definitions and retrieval configuration — for GenAI applications, these encode business logic.
2. Protecting the artefact
| Where | Controls |
|---|---|
| Registry and storage | Encryption with dedicated keys, least-privilege access, download logging, alerts on bulk or unusual downloads |
| In transit | TLS, signed artefacts, private network paths |
| Serving hosts | Hardened, isolated hosts; weights loaded into memory from encrypted storage; no interactive access for most staff |
| Edge and on-device | Model encryption and obfuscation, secure enclaves where available — and the assumption that determined attackers may still extract it |
| Partners | Confidential computing with attestation (Module 3), contractual protections |
| People | Insider-risk controls for staff with access; offboarding that revokes registry access |
Treat bulk model downloads like bulk data exfiltration: they should be rare, approved and monitored.
3. Model extraction
Model extraction (model stealing) recreates a model's functionality by querying it and training a copy — a surrogate — on the inputs and outputs. The attacker never touches the weights.
Why it matters:
- IP loss — a competitor gets similar capability without the investment.
- Attack staging — a local surrogate lets attackers craft adversarial examples offline, which often transfer to the original.
- Distillation abuse — using a commercial model's outputs to train a competing model, often against terms of service.
Signals and controls:
- Rate limits and quotas per identity and per tenant; tiered access for high volumes.
- Query-pattern monitoring — systematic coverage of the input space, synthetic-looking inputs, very high volumes from new accounts.
- Output minimisation — return labels rather than full probability vectors where possible; round confidence scores.
- Authentication and terms — no anonymous high-volume access; contractual prohibition of training on outputs.
- Watermarking and fingerprinting — to prove a suspected copy came from your model.
4. Watermarking and fingerprinting
| Technique | What it marks | Purpose |
|---|---|---|
| Model watermarking | The model's parameters or behaviour (e.g. secret trigger inputs that produce a known response) | Prove ownership of a stolen or extracted model |
| Model fingerprinting | Unique, naturally occurring characteristics of a model's behaviour | Identify whether a suspect model is derived from yours without modifying it |
| Output watermarking | Generated content (statistical patterns in text, signals in images or audio) | Identify AI-generated content and its source model |
Limits: watermarks can be weakened by fine-tuning, pruning, distillation or paraphrasing; detection is probabilistic. They support legal and investigative work but don't prevent theft.
Worked example — spotting an extraction campaign
A document-classification API shows the following over a weekend:
| Signal | Observation |
|---|---|
| Volume | Three new accounts, 180,000 queries in 48 hours (normal: < 2,000 per account per day) |
| Inputs | Templated, synthetic-looking documents systematically varying one field at a time |
| Outputs requested | Full probability vectors via an undocumented parameter |
| Payment | Trial credits, no verified organisation |
Response: throttle and suspend the accounts; remove probability vectors from the public tier; require verified organisations for high quotas; add an alert on input-space coverage patterns. Watermark trigger queries are scheduled against a competitor's newly launched classifier to check for derivation.
🔐 Security lens
System prompts and tool definitions are part of your IP and your attack surface. Assume they can be extracted, keep secrets out of them, and monitor for leaks of their distinctive wording.
Key takeaways
- Protect weights, architecture, code, data and GenAI configuration — the artefact and the behaviour.
- Secure weights with encryption, access control, isolation, download monitoring and, at partners, attested confidential computing.
- Extraction trains a surrogate through queries; counter with rate limits, pattern monitoring, output minimisation and terms.
- Watermarking proves ownership; fingerprinting identifies derivation; output watermarking marks generated content — all probabilistic and removable.
Self-check
- How can an attacker steal a model without ever accessing its weights?
- Why does returning full probability vectors make extraction easier?
- What is the difference between model watermarking and output watermarking?