3.5 Model lifecycle and security governance
In this lesson you will:
- Name the owners of a model and scale the approval evidence to its risk tier.
- Use a model card and a model registry, and turn the registry into a control that actually blocks unapproved models.
- Secure the path from development to production with gates, safe file formats and signatures.
- Sanitize model inputs and outputs, release changes gradually, and roll back safely.
- Monitor models in production and keep audit trails that stand up in an investigation.
Opening case: the Friday hotfix
Sentinel's recall has been slipping, and the fraud team is under pressure. On a Friday afternoon, a senior data scientist downloads a fraud-detection model from a public model hub that "looks better", fine-tunes it on her laptop with last month's data, and gets impressive numbers. She saves it as a
.pklfile, copies it to the production model server using a service account whose password she knows, and restarts the scoring service. Nobody else is involved.Over the weekend, Sentinel flags 18% of all card payments as fraud. Thousands of customers have their cards blocked. On Monday, the team cannot quickly restore the previous model, because nobody is sure which file was running before. The security team also discovers that the downloaded model file contained code that opened a network connection to an unknown address when it was loaded.
By the end of this section you will be able to list every lifecycle control that was bypassed, and explain how a model registry, release gates, safe formats, signing and staged rollout would have stopped this at four different points.
Ownership, risk tiers and approval
Every model in production needs a named model owner, accountable for its performance, documentation and monitoring, and a business owner, accountable for its use. Models are classified by risk, usually inheriting the tier of the use (section 2.3), and the approval workflow is proportionate to that tier [5][6]. Figure 3.16 shows the lifecycle gates and the evidence each requires.
| Gate | Evidence required | Who signs at tier 3 (high risk) |
|---|---|---|
| Develop | Model card draft, lineage, AIBOM | Model owner |
| Validate | Performance, fairness, robustness and security results | Independent validator |
| Approve | Approval record with conditions | AI governance committee |
| Production | Signed artifact in the approved registry; monitoring set up | Release manager, automated gate |
| Monitor | Reports, alerts, reassessments | Model owner |
| Retire | Revocation and disposal evidence | Model owner and data owner |
A low-risk internal model may need only peer review and registration. Sentinel is high-risk for the bank's customers and finances, so it needs independent validation and committee approval. In the opening case, one person developed, validated, approved and deployed the model.
Model cards and the registry
A model card documents a model's purpose, intended and out-of-scope uses, training data summary, evaluation results (including across relevant groups), limitations, security testing and contacts (section 2.6). It is written for the people deciding whether and how to use the model, and it is updated with every version. Figure 3.17 shows one for a fraud model.
A model registry is the catalog of approved models and their versions, with metadata, artifact hashes, lineage, approval status and owner. Its security value comes from making it an enforceable deployment control:
The deployment pipeline will deploy only a model version that is in the registry, approved for its tier, and whose artifact hash and signature match the registered record.
With that rule enforced, a model copied in from a laptop or downloaded straight from a public hub cannot reach production, however senior the person copying it. A registry that is merely a list people are asked to keep up to date is documentation; a registry the pipeline checks is a control.
Securing the path to production
Figure 3.18 shows how signing protects a model from training to load.
The security controls on the release path are [6][7][9]:
- Validation gates: automated checks for performance against thresholds (including on important slices, not just averages), fairness metrics, robustness, security tests (including prompt-injection and jailbreak tests for LLMs), and scans of the model artifact.
- Safe packaging: some model file formats, notably Python's pickle (
.pkl, and formats built on it), can execute arbitrary code when loaded. A malicious model file can therefore compromise whatever machine opens it, which is exactly what happened in the opening case. Prefer formats that store only data, such as safetensors [30]; scan artifacts from outside sources; and load untrusted models only in isolated environments. - Signing: hash and sign model artifacts (weights, tokenizer, configuration, adapters) at approval, and verify the signature and hash at load time, using tooling such as Sigstore-based model signing [29]. A tampered or unregistered artifact fails verification and is refused.
- Version and protect everything that shapes behavior: not just weights, but system prompts, tool definitions, guardrail configuration and retrieval indexes.
- Hardened serving: inference servers with minimal privileges, patched dependencies, network segmentation, authenticated endpoints and no interactive logins.
- Pipeline credentials: the credentials that can publish models are high-value targets. Use short-lived, narrowly scoped credentials issued to the pipeline, not shared passwords known to individuals (Module 4).
Exam lens. "A model downloaded from a public hub in pickle format" points to arbitrary code execution on load. The best answers are safe formats (safetensors), scanning, and loading untrusted artifacts in isolation, not "check it for bias".
Input and output sanitization
At runtime, treat the model as a component that receives untrusted input and produces untrusted output.
- Sanitize inputs: validate type, length and format; filter known attack patterns; strip hidden content such as invisible Unicode characters and unexpected markup.
- Sanitize outputs before they are displayed or passed on: escape HTML to prevent cross-site scripting, validate structured output against a schema, filter personal data, and never pass model output directly to a shell, database query or interpreter.
The principle is simple: model output is untrusted input to whatever consumes it.
Release and rollback
Release model updates gradually, as Figure 3.19 shows.
- Shadow deployment: the new model scores live traffic alongside the old one, but its decisions are not used. Compare the two.
- Canary: route a small share of traffic (for example 5%) to the new model.
- Widen to 25%, then 100%, only while guardrail metrics hold.
- Roll back automatically to the last approved version if any metric breaches its threshold.
Worked example: guardrail metrics for Sentinel
| Metric | Normal range | Rollback threshold |
|---|---|---|
| Share of payments flagged | 0.3% to 0.6% | Above 1.5% |
| Recall on daily labeled sample | Above 75% | Below 65% |
| Customer complaints about blocked cards | Baseline | Twice baseline |
| Scoring latency (p99) | Under 80 ms | Over 200 ms |
In shadow mode, the Friday model would have flagged 18% of payments while the production model flagged 0.4%: a difference visible within minutes, with zero customer impact. Even as a 5% canary, the breach of the 1.5% threshold would have triggered automatic rollback within an hour. Keep previous approved versions available in the registry so rollback is a configuration change, not a search for files.
Monitoring and audit trails
Production models are monitored for:
- performance and drift (section 1.3);
- fairness across groups;
- security anomalies, such as unusual query patterns suggesting extraction, spikes in refused or filtered requests, or sudden changes in output distribution.
Results feed back to governance: a drift alert triggers reassessment, and repeated issues may lead to retraining or retirement.
Keep audit trails of AI data and model events: dataset approvals, training runs and the data versions they used, registrations, approvals, deployments, configuration changes, access to sensitive datasets and weights, and rollbacks. Logs should be tamper-evident and retained long enough for investigations and regulatory obligations, but they must not become a privacy problem themselves: mask personal data and sensitive prompt content where full content is not needed (Module 5).
The opening case, resolved
| Control bypassed | What should have happened |
|---|---|
| Ownership and separation of duties | One person developed, validated, approved and deployed. Tier 3 requires independent validation and committee approval |
| Supply-chain intake | A model from a public hub must go through intake: provenance, license, scanning (section 3.7) |
| Safe format | Pickle executed code on load. Use safetensors; scan and load untrusted models in isolation |
| Registry as a gate | The pipeline should refuse any artifact not registered, approved and matching its signed hash |
| Credentials | A shared service-account password let a person deploy by hand. Only the pipeline should hold short-lived deployment credentials |
| Staged release | Shadow mode or a 5% canary would have revealed the 18% flag rate before customers were affected |
| Rollback | Previous approved versions in the registry make rollback a one-step change |
| Audit trail | Deployment records would have shown exactly what was running and when |
The four points at which the model would have been stopped: intake (scan and reject the pickle), registry gate (not registered or approved), signature verification at load (no valid signature), and canary metrics (automatic rollback).
Check your understanding
Try each before reading the reasoning.
1. A team keeps a spreadsheet listing every production model, its owner and its version. Is that a model registry control?
Reasoning: it is documentation, not a control. A registry becomes a control when the deployment pipeline enforces it: only registered, approved artifacts whose hash and signature match can be deployed.
2. An LLM-based assistant generates HTML that is shown in a web page. A tester gets it to output a script tag that runs in other users' browsers. Which control failed?
Reasoning: output sanitization. Model output must be treated as untrusted: escape or sanitize it before rendering, and validate it against what the page expects.
3. A new model version passes all offline tests. What is the safest way to put it into production for a high-impact use?
Reasoning: a staged release: shadow mode first, then a small canary with predefined guardrail metrics and automatic rollback, widening only while metrics hold, with the previous approved version ready in the registry.
Key takeaways
- Every model has a model owner and a business owner. Approval evidence and independence scale with risk tier.
- Model cards describe the model for the people who use it; the registry records what is approved.
- The registry is a control only when the pipeline deploys nothing else.
- Pickle can execute code on load. Prefer safetensors, scan artifacts, and isolate untrusted models.
- Hash and sign everything that shapes behavior, and verify at load.
- Model output is untrusted input to whatever consumes it.
- Release with shadow mode and canaries, guardrail metrics and automatic rollback.
- Monitor performance, fairness and security signals, and keep tamper-evident, privacy-aware audit trails.