AI Threats, Testing and Lifecycle Management, 60 min read

6.2 Secure AI development lifecycle and change management

In this lesson you will:

  1. Say which security activity and which automated gate belongs at each stage of building an AI system.
  2. Keep experiments, training and production apart without slowing teams down.
  3. Recognize every kind of change that alters AI behavior, and control all of them.
  4. Release AI changes gradually, and roll back cleanly when needed.

Opening case: the rollback that did not roll back

On Monday, Meridian released version 3.4 of its customer chatbot. The release changed three things: a newer foundation model version, a rewritten system prompt, and a new tool that lets the chatbot look up a customer's card status.

By Wednesday, complaints showed the chatbot was telling some customers their card was blocked when it was not. The team rolled back "to 3.3" by switching the model endpoint back to the old version. The problem continued. On Thursday they found that the new system prompt and the card-status tool were still live. The tool was returning stale data, and the new prompt told the model to "always state the card status first".

Nobody could say exactly what "version 3.3" had consisted of.

By the end of this section you will be able to define what a release of an AI system is, so that a rollback restores all of it, and design a rollout that would have caught the problem on Monday afternoon.

Security in every stage

A secure AI development lifecycle (AIDLC) extends the secure software development lifecycle to AI [6][19]. NIST SP 800-218A, the SSDF community profile for generative AI and dual-use foundation models, adds AI-specific practices to the Secure Software Development Framework. Figure 6.5 shows the stages.

Secure AI development lifecycle with security activities per phase
Figure 6.5 Secure AI development lifecycle with security activities per phase
Stage Security activities Automated gates (examples)
Plan and design Use-case risk tiering, threat modeling, privacy and impact assessments, security requirements Inventory entry and risk tier required before work starts
Data Provenance and license checks, personal-data scanning, poisoning checks, lineage Dataset must have a signed manifest and passed validation
Build and train Isolated environments, pinned dependencies, secure configuration, experiment tracking Dependency and secret scanning; reproducible build
Evaluate Performance, fairness, robustness, safety and security tests; red teaming for higher tiers Thresholds for each metric; jailbreak and injection test pass rates
Deploy Signed artifacts, registry approval, hardened serving, guardrails, monitoring Signature and approval verification; policy-as-code checks
Operate Monitoring, incident response, drift detection, reassessment triggers Automatic rollback on metric breaches
Retire Secure decommissioning, data and model disposal, inventory update Credentials revoked; endpoints removed

The same structure applies to:

  • MLOps, where you train your own models;
  • LLMOps, where you build applications on foundation models. Here the "build" stage is often prompt engineering, retrieval design and tool integration rather than training, and those artifacts need the same rigor.

The card-status tool in the opening case went through no "evaluate" gate of its own. It was treated as part of the prompt work, not as a new integration with its own data-quality risk.

Environment separation and secure integration

Keep research, development, training and production environments separate, with progressively stricter controls [9]:

Environment Purpose Key controls
Research Exploration Synthetic, public or approved de-identified data; no production credentials
Development Build and test applications Test data; non-production model endpoints
Training Train or fine-tune on sensitive data Isolated; no internet access; tightly controlled egress
Production Serve users Changed only through the pipeline; no interactive access by default

Apply least privilege to each environment's identities. Integrate with external model APIs securely:

  • credentials from a vault, never in code or notebooks;
  • private connectivity where available;
  • version pinning, so the provider cannot change your model silently;
  • data sent only as the environment's classification allows.

Change management for everything that shapes behavior

In traditional software, a change is a code change. In AI, behavior also changes when any of these change, as Figure 6.6 shows:

  • the model or its version;
  • the datasets used to train or fine-tune it;
  • the retrieval sources, chunking or index;
  • the prompts (system prompts, templates);
  • the guardrails (rules, classifier models, thresholds);
  • the tools (which tools exist, their permissions, their definitions).
Types of AI changes and their typical review depth
Figure 6.6 Types of AI changes and their typical review depth

All of them must be under change control: versioned, reviewed, tested and approved in proportion to their risk.

The release manifest

Bundle them into a release manifest: a record of the exact version of every component that together make up one release of the AI system. The manifest is what gets tested, approved, deployed and, if needed, rolled back. Figure 6.7 shows one.

A release manifest bundles every behavior-shaping component, and rollback restores the whole bundle
Figure 6.7 A release manifest bundles every behavior-shaping component, and rollback restores the whole bundle

Worked example: what 3.3 and 3.4 should have been

Component Release 3.3 Release 3.4
Foundation model provider-model 2026-05 (pinned) provider-model 2026-09 (pinned)
System prompt prompt v21 (hash 9e1c…) prompt v24 (hash 4ab7…)
Guardrail config guard v8 guard v8
Tools faq_search v3, branch_finder v2 faq_search v3, branch_finder v2, card_status v1
Retrieval index faq-index 2026-10-01 faq-index 2026-10-01
Evaluation results eval run 3.3 (passed) eval run 3.4 (passed)

With this manifest, "roll back to 3.3" restores the model, prompt v21 and removes card_status. A rollback that restores the old model but keeps the new system prompt is not a true rollback.

Security lens. Ask for the release manifest of any AI system you assess. If the team cannot produce one, they cannot tell you what is running, what was tested, or what a rollback will restore.

Staged rollouts

Release AI changes gradually, because offline testing never captures all real-world behavior [8]:

  • Shadow deployment: the new version receives a copy of real traffic. Its outputs are recorded and compared, but users see only the old version's results.
  • Canary deployment: a small share of users or traffic goes to the new version, increasing in steps if metrics stay healthy (section 3.5).
  • Pre-agreed triggers: before release, agree the metrics and thresholds that will stop the rollout or trigger rollback. Examples: harmful-output rate, refusal rate, guardrail triggers, error rate, latency, cost, user complaints.
  • Rollback of the whole bundle: restore every component in the previous release manifest, not just the model.

For release 3.4, a shadow run comparing card-status statements with the card system of record would have shown the mismatch before any customer saw it. A canary at 5% with a trigger on "card-status complaints per thousand conversations" would have limited the harm to a few hours.

Check your understanding

Try each before reading the reasoning.

1. A team updates only the retrieval chunking strategy and says no change approval is needed because "the model did not change". How should you respond?

Reasoning: chunking changes what the model sees and therefore its behavior. Retrieval is one of the components under change control. Version it, test it and approve it in proportion to risk, as part of a new release manifest.

2. A data scientist wants production database credentials in the research notebook environment "to save time". What principle does this break, and what is the alternative?

Reasoning: environment separation and least privilege. Research uses synthetic, public or approved de-identified data with no production credentials. Provide an approved de-identified extract instead.

3. What is the difference between shadow and canary deployment?

Reasoning: in shadow, the new version processes copied traffic but users see only the old version's results; in canary, a small share of real users actually receive the new version's results.

Key takeaways

  • A secure AIDLC puts security activities and automated gates at every stage, for MLOps and LLMOps alike. NIST SP 800-218A adds AI practices to the SSDF.
  • Separate research, development, training and production, with least privilege, vaulted credentials, private connectivity and version pinning for external models.
  • Models, data, retrieval, prompts, guardrails and tools all change behavior. Version and control all of them, bundled in a release manifest.
  • Roll out with shadow and canary stages and pre-agreed triggers, and roll back the whole bundle.