Model Security & the Secure AI Lifecycle, 25 min read

Model confidentiality: weights, extraction and watermarking

In this lesson you will:

  • Protect model weights, algorithms and related intellectual property
  • Explain model extraction attacks and the controls that prevent them
  • Distinguish model watermarking from fingerprinting and output watermarking

A trained model can represent months of work, unique data and significant compute cost — and it is a file that can be copied. Confidentiality controls protect it from theft through two routes: stealing the artefact and stealing the behaviour.

1. What needs protecting

  • Weights and checkpoints — the model itself.
  • Architecture and hyperparameters — design know-how.
  • Training code and pipelines — how it was made.
  • Proprietary training data — often the real competitive advantage.
  • System prompts, tool definitions and retrieval configuration — for GenAI applications, these encode business logic.

2. Protecting the artefact

Controls protecting model weights across storage, transfer and runtime.
Figure 5.3 Protecting model weights wherever they exist — in the registry, in transit, on serving hosts and at the edge.
Where Controls
Registry and storage Encryption with dedicated keys, least-privilege access, download logging, alerts on bulk or unusual downloads
In transit TLS, signed artefacts, private network paths
Serving hosts Hardened, isolated hosts; weights loaded into memory from encrypted storage; no interactive access for most staff
Edge and on-device Model encryption and obfuscation, secure enclaves where available — and the assumption that determined attackers may still extract it
Partners Confidential computing with attestation (Module 3), contractual protections
People Insider-risk controls for staff with access; offboarding that revokes registry access

Treat bulk model downloads like bulk data exfiltration: they should be rare, approved and monitored.

3. Model extraction

Model extraction (model stealing) recreates a model's functionality by querying it and training a copy — a surrogate — on the inputs and outputs. The attacker never touches the weights.

Model extraction: attacker queries the API, collects outputs and trains a surrogate.
Figure 5.4 Model extraction via an API. The surrogate can then be used directly or to craft attacks against the original.

Why it matters:

  • IP loss — a competitor gets similar capability without the investment.
  • Attack staging — a local surrogate lets attackers craft adversarial examples offline, which often transfer to the original.
  • Distillation abuse — using a commercial model's outputs to train a competing model, often against terms of service.

Signals and controls:

  • Rate limits and quotas per identity and per tenant; tiered access for high volumes.
  • Query-pattern monitoring — systematic coverage of the input space, synthetic-looking inputs, very high volumes from new accounts.
  • Output minimisation — return labels rather than full probability vectors where possible; round confidence scores.
  • Authentication and terms — no anonymous high-volume access; contractual prohibition of training on outputs.
  • Watermarking and fingerprinting — to prove a suspected copy came from your model.

4. Watermarking and fingerprinting

Technique What it marks Purpose
Model watermarking The model's parameters or behaviour (e.g. secret trigger inputs that produce a known response) Prove ownership of a stolen or extracted model
Model fingerprinting Unique, naturally occurring characteristics of a model's behaviour Identify whether a suspect model is derived from yours without modifying it
Output watermarking Generated content (statistical patterns in text, signals in images or audio) Identify AI-generated content and its source model

Limits: watermarks can be weakened by fine-tuning, pruning, distillation or paraphrasing; detection is probabilistic. They support legal and investigative work but don't prevent theft.

Worked example — spotting an extraction campaign

A document-classification API shows the following over a weekend:

Signal Observation
Volume Three new accounts, 180,000 queries in 48 hours (normal: < 2,000 per account per day)
Inputs Templated, synthetic-looking documents systematically varying one field at a time
Outputs requested Full probability vectors via an undocumented parameter
Payment Trial credits, no verified organisation

Response: throttle and suspend the accounts; remove probability vectors from the public tier; require verified organisations for high quotas; add an alert on input-space coverage patterns. Watermark trigger queries are scheduled against a competitor's newly launched classifier to check for derivation.

🔐 Security lens

System prompts and tool definitions are part of your IP and your attack surface. Assume they can be extracted, keep secrets out of them, and monitor for leaks of their distinctive wording.

Key takeaways

  • Protect weights, architecture, code, data and GenAI configuration — the artefact and the behaviour.
  • Secure weights with encryption, access control, isolation, download monitoring and, at partners, attested confidential computing.
  • Extraction trains a surrogate through queries; counter with rate limits, pattern monitoring, output minimisation and terms.
  • Watermarking proves ownership; fingerprinting identifies derivation; output watermarking marks generated content — all probabilistic and removable.

Self-check

  1. How can an attacker steal a model without ever accessing its weights?
  2. Why does returning full probability vectors make extraction easier?
  3. What is the difference between model watermarking and output watermarking?