AI Data Management, Privacy and Model Security, 65 min read

3.6 Model confidentiality, integrity and availability

In this lesson you will:

  1. Say what a model's intellectual property consists of, and protect the weights wherever they are.
  2. Explain how attackers copy a model through its API, how to spot it, and how to make it harder.
  3. Tell model watermarking from fingerprinting, and say which proves what.
  4. Make sure the model you are running is the model you approved.
  5. Keep AI services available and affordable under attack, and make them fail safely.

Opening case: the customer who asked too many questions

Meridian now sells Sentinel API, its fraud-scoring model, as a service to 40 smaller banks and payment firms. Each customer sends transaction details and receives a fraud probability with four decimal places and the top three contributing factors.

One new customer, a payments start-up, sends 2.4 million scoring requests in its first month, about 60 times more than customers of its size. Many requests are near-identical transactions with one field changed at a time. Three months later, the start-up launches its own fraud-scoring product, with behavior on unusual transactions that looks remarkably like Sentinel's.

In the same quarter, Meridian's internal LLM assistant runs up a model-provider bill of USD 310,000 in one week, against a normal USD 22,000, after an internal script loops on a very long document.

By the end of this section you will be able to name what the start-up probably did, how Meridian could have detected and limited it, how it could prove copying, and how to stop a runaway cost spike from becoming an outage.

Confidentiality: protecting model intellectual property

A trained model can represent months of work and large compute bills. Its intellectual property includes:

  • the weights;
  • the architecture and hyperparameters;
  • the training data and recipe;
  • fine-tuning adapters;
  • the system prompts and configuration that shape its behavior.

Protect the weights wherever they are, as Figure 3.20 shows [3][7].

Controls protecting model weights in storage, in transit, on serving hosts, on edge devices and at partner sites
Figure 3.20 Controls protecting model weights in storage, in transit, on serving hosts, on edge devices and at partner sites
  • Registry and storage: encryption with organization-controlled keys, strict access control with a separate role for reading weights, and logging of every download.
  • In transit: TLS, signed artifacts, private network paths.
  • Serving hosts: isolated and hardened, with no interactive access, so users reach the model only through an API, never the files.
  • Edge and devices: encrypt and obfuscate, but assume the weights can be extracted from any device you give to someone else.
  • Partner sites: trusted execution environments with attestation, plus contract terms.
  • People: insider-risk controls, least-privilege registry roles, and offboarding that revokes access.

Treat a bulk model download like a bulk data exfiltration: rare, approved and monitored.

Model extraction

Model extraction (or model stealing) recreates a functional copy of a model by querying it and training a substitute on its answers [23]. The attacker never touches the weights. They use the API like any customer, but at volume and with chosen inputs, as Figure 3.21 shows.

Model extraction by querying an API and training a surrogate, with signals and controls
Figure 3.21 Model extraction by querying an API and training a surrogate, with signals and controls

A stolen copy can be used for free, studied offline to find evasion attacks (for example, which transactions slip past fraud scoring), or used to infer training data.

Worked example: the signals in Sentinel API's logs

Signal Typical customer The start-up Why it matters
Monthly queries relative to its transaction volume About 1× (one score per real payment) About 60× Scoring far more transactions than it processes
Share of near-duplicate queries (one field changed) Under 2% 71% Systematically probing how each input moves the score
Coverage of unusual input values (rare countries, extreme amounts) Matches real traffic Uniform across the whole input space Synthetic inputs designed to map the model, not real payments
Use of returned explanations Rarely retrieved Retrieved on every call Harvesting the model's reasoning

Each signal on its own could be innocent; together they are the fingerprint of extraction. Extraction risk is reduced, though never eliminated, by:

  • Access control: authenticate API users and know your customers; require verified accounts for high volumes.
  • Output minimization: return only what is needed, such as a risk band ("low / medium / high") rather than a four-decimal probability, and explanations only where entitled. Precise scores and factor lists make extraction far cheaper (section 2.2).
  • Rate limits and quotas per customer and per key, sized to legitimate use (for example, tied to the customer's declared transaction volume).
  • Detection: monitor for the patterns above: systematic coverage of the input space, near-duplicate probing and volumes out of proportion to the customer.
  • Contractual terms that prohibit using outputs to train competing models, with audit rights.

Watermarking and fingerprinting

Two techniques help prove ownership or trace a leak, and they are easily confused. Figure 3.22 contrasts them.

Model watermarking is embedded deliberately; fingerprinting measures existing traits
Figure 3.22 Model watermarking is embedded deliberately; fingerprinting measures existing traits
  • Model watermarking embeds a hidden, verifiable signal into a model or its outputs on purpose. For example, a model can be trained to give an unusual, secret response to a set of trigger inputs, or a language model can bias its word choices in a statistically detectable way [34]. The owner can later show the signal in a suspect model or text as evidence.
  • Model fingerprinting identifies a model by its existing, natural characteristics, such as how it responds to specially chosen inputs, without changing it. It can be applied to models already deployed.

Watermarks are active (put in deliberately); fingerprints are passive (measured). Giving each licensee a uniquely watermarked copy lets the owner trace which copy leaked. Neither is foolproof: fine-tuning, pruning or paraphrasing can weaken them.

For Sentinel API, Meridian could have trained Sentinel to give a distinctive, otherwise implausible score to a small set of secret trigger transactions. If the start-up's model gives the same distinctive scores to the same secret triggers, that is strong evidence it was trained on Sentinel's outputs. A surrogate trained on extracted outputs often inherits such triggers.

Exam lens. "Prove a leaked model is ours" or "find which licensee leaked it" points to watermarking (unique per licensee for the second). "Identify which model produced this behavior, without having modified it" points to fingerprinting.

Integrity: making sure the model is the model

A swapped or modified model file is a powerful attack: an attacker who replaces the weights controls every output. Protect integrity with [9][29][30]:

  • hashing every model artifact and recording the hash in the registry;
  • digital signatures on artifacts, verified before loading;
  • safe formats that cannot execute code when loaded, such as safetensors rather than pickle;
  • verification points at every handover: registration, the pull into the deployment pipeline, and the start of the serving container;
  • version control for models, adapters, prompts and configuration, so any change is visible, attributable and reversible.

Section 3.5 showed these as release controls. Here the point is runtime assurance: at any moment, the serving system should be able to prove that the artifact it has loaded matches the approved, signed version in the registry.

Availability: keeping AI services up and affordable

AI availability attacks often aim at cost as much as uptime, because each request consumes expensive compute. OWASP calls this unbounded consumption [32]. A runaway internal script and a malicious flood look the same on the bill.

Worked example: anatomy of a USD 310,000 week

Factor Normal During the spike
Requests per day 40,000 55,000
Average tokens per request (input + output) 1,800 120,000 (a 300-page document resent on every loop)
Tokens per day 72 million 6.6 billion
Request-count rate limit 100 per minute per user Never breached

The request rate barely changed, so a request-count limit never triggered. The cost came from tokens per request. That is why limits must be token-aware.

Controls:

  • Token-aware rate limits: limit tokens, not just requests.
  • Quotas and budget caps per user, application and tenant, with alerts at 50%, 80% and 100% of budget and a hard stop or approval step beyond it.
  • Input and output size limits, and caps on the number of agent steps or tool calls per task (a looping agent is a cost attack on yourself).
  • Graceful degradation: when limits are reached or a provider is down, fall back to a smaller model, a cached answer, a rules engine or a human channel rather than failing outright, as Figure 3.23 shows.
  • Fail safe: under stress, the system should default to the safe state (for example, declining to act, or holding a payment for review) rather than skipping safety checks to keep up.
A graceful degradation path from the primary model through smaller models, caches and rules to a human queue
Figure 3.23 A graceful degradation path from the primary model through smaller models, caches and rules to a human queue

For Sentinel API, fail-safe has a specific meaning: if the scoring service is overloaded, payments should be held for review or scored by the rules engine, not waved through unscored. An attacker who can overload a fraud model must not be able to turn it off.

The opening case, resolved

  • What the start-up probably did: model extraction: 2.4 million mostly synthetic, near-duplicate queries with harvested explanations, used to train a surrogate.
  • How Meridian could have detected it: volume out of proportion to the customer's business, 71% near-duplicate probing, uniform coverage of rare input values, and constant retrieval of explanations, ideally alerting within days.
  • How Meridian could have limited it: quotas tied to declared transaction volume, risk bands instead of four-decimal probabilities, explanations only where contractually needed, verified onboarding, and contract terms forbidding training on outputs.
  • How Meridian could prove copying: secret watermark triggers in Sentinel's behavior (or a unique watermark per customer) that reappear in the start-up's model.
  • The USD 310,000 week: token-aware limits, per-application budget caps with alerts and hard stops, input size limits and loop caps would have stopped it in hours. Graceful degradation and fail-safe design keep a cost control from becoming an outage or a security gap.

Check your understanding

Try each before reading the reasoning.

1. A company distributes a speech model inside a mobile app. How should it think about the confidentiality of the weights?

Reasoning: assume the weights can be extracted from any device in a user's hands. Obfuscate and encrypt to raise the cost, watermark the model to prove ownership, keep the most valuable capabilities server-side, and rely on contract and legal terms.

2. An LLM API has a limit of 60 requests per minute per user, yet costs have tripled. What is the most likely gap?

Reasoning: limits counted in requests rather than tokens. Long prompts or outputs can multiply cost without increasing request counts. Add token-aware limits, size caps and budget alerts.

3. Under heavy load, a fraud-scoring service is configured to approve transactions without scoring to keep payments flowing. What principle does this violate?

Reasoning: fail safe. Under stress the system should default to the safe state, such as holding for review or falling back to deterministic rules, not skip the control. Otherwise an attacker can disable fraud detection by overloading it.

Key takeaways

  • Model IP includes weights, architecture, training recipe, adapters and system prompts. Protect weights in storage, transit, serving, edge and partner sites, and assume edge copies can be extracted.
  • Model extraction uses the API, not the files. Watch volume, near-duplicate probing and input-space coverage; minimize outputs; set quotas; forbid training on outputs by contract.
  • Watermarking is deliberate and can be unique per licensee; fingerprinting measures existing traits.
  • Hashes, signatures, safe formats and verification at every handover keep the running model the approved model.
  • Availability attacks target cost: use token-aware limits, budget caps, size and loop caps, graceful degradation, and fail-safe defaults.