Model Security & the Secure AI Lifecycle, 25 min read

Model governance, model cards and approved registries

In this lesson you will:

  • Define model ownership, risk classification and approval workflows
  • Write and use model cards as governance artefacts
  • Operate an approved model registry as a security control
  • Monitor models across their security lifecycle

Module 2 governed AI use cases. This module zooms in on the model itself — an asset that is valuable, fragile and easy to copy. Governance starts with knowing which models you have, who owns each, and which are allowed into production.

1. Model governance essentials

Element What it means
Ownership Every model has a named owner accountable for its performance, risk and retirement — plus a technical owner who maintains it
Risk classification Each model inherits the highest risk tier of the use cases it serves; a general model used in a Tier 3 decision is a Tier 3 model
Approval workflow Defined gates — development → validation → approval → production — with evidence required at each
Independent validation For higher tiers, someone other than the developer validates performance, robustness and fairness (the model-risk-management discipline used in banking)
Inventory linkage Model records link to use-case records, datasets (lineage) and AIBOMs
Model approval workflow with gates and required evidence.
Figure 5.1 A model approval workflow. Evidence required at each gate scales with risk tier.

2. Model cards

A model card is a short, structured document describing a model: what it is for, how it was built and evaluated, and where it should not be used. Originally proposed as a transparency tool, it is now a core governance artefact — and the EU AI Act's technical-documentation duties cover much of the same ground.

Example model card sections for a fraud model.
Figure 5.2 A model card. Security teams care most about intended use, out-of-scope uses, training data, evaluations (including robustness) and known limitations.

Typical sections:

  1. Model details — name, version, owner, type, date, licence.
  2. Intended use — primary uses and users; out-of-scope uses.
  3. Training data — sources, dates, preprocessing, known gaps (link to datasheets).
  4. Evaluation — metrics, test sets, results overall and by subgroup.
  5. Robustness and security — adversarial testing performed, known weaknesses.
  6. Ethical considerations and limitations — bias findings, failure modes, caveats.
  7. Operational information — dependencies, hardware, monitoring thresholds, retraining cadence.

For generative systems, providers also publish system cards covering the whole system — model plus safeguards, tools and policies.

3. Approved model registries

A model registry stores model artefacts with their versions, metadata and status. An approved registry adds a control: production systems may only load models whose status is approved.

Registry security features:

  • Status and stage — development, staging, approved, deprecated, revoked.
  • Immutable versions — an approved version cannot be overwritten; changes create new versions.
  • Integrity metadata — hash and signature for every artefact (Lesson 5.3).
  • Access control — who can register, promote and download; promotion to approved requires a different role from the developer.
  • Lineage links — training data versions, code commit, training run, evaluation results.
  • Revocation — a revoked model is blocked at load time across all environments.

Third-party and open models enter the registry only after the provenance checks in Module 3 — making the registry the single gate for all models, built or bought.

4. Security lifecycle monitoring

After approval, models need watching:

  • Performance and drift — accuracy against fresh labels, input distribution shifts, output distribution shifts.
  • Bias tracking — fairness metrics by subgroup over time.
  • Anomaly detection — unusual input patterns (possible evasion or extraction attempts), unusual query volumes, spikes in low-confidence outputs.
  • Security signals — guardrail triggers, injection attempts, abuse reports.

Thresholds breached → alert the owner → investigate → retrain, roll back or restrict.

Worked example — a registry stops a risky shortcut

A developer under deadline pressure wants to deploy a newly fine-tuned credit model straight from a notebook.

Control What happened
Production loader Refuses to load any model not marked approved in the registry
Registry promotion Developer can register the model, but promotion needs the model-validation role
Validation Independent validator finds approval rates for one region dropped 12 points versus the current model
Model card Updated with the finding; out-of-scope note added
Outcome Model held in staging; data issue traced to a missing region in the fine-tuning set; fixed and re-validated a week later

🔐 Security lens

Enforce the registry at load time in production (allow-list by hash and status), not just by policy. Otherwise a model file copied onto a server bypasses every governance gate.

Key takeaways

  • Every model needs an owner, a risk classification inherited from its use cases, and an approval workflow with independent validation for higher tiers.
  • Model cards document intended and out-of-scope use, data, evaluations, robustness and limitations.
  • An approved registry with immutable versions, integrity metadata, separation of duties and revocation is the single gate for built and bought models.
  • Monitor drift, bias, anomalies and security signals after approval.

Self-check

  1. A general-purpose model serves both a Tier 1 and a Tier 3 use case. What tier is the model?
  2. Which model-card section should state that a model must not be used for employment decisions?
  3. Why should the developer not be able to promote their own model to approved?