Secure AI Architecture, Engineering and IAM, 60 min read

4.3 Securing MLOps and LLMOps pipelines

In this lesson you will:

  1. Name the threat at each stage of a model pipeline.
  2. Harden the build and training system so that an attacker cannot persist in it or steal long-lived secrets from it.
  3. Organize data ingestion into trust zones.
  4. Capture lineage automatically, and use it when something goes wrong.
  5. Turn quality and security checks into gates that cannot be skipped.
  6. Build the ability to forget data into the pipeline from the start.

MLOps applies DevOps practices (automation, version control, continuous integration and delivery) to building and running machine learning models. LLMOps extends it to LLM applications, where the artifacts include prompts, retrieval indexes, guardrail configurations and tool definitions as well as models. The pipeline that builds and ships these artifacts is a prime target: whoever controls the pipeline controls what reaches production [6][7].

Opening case: the gate that wasn't

An attacker phishes a Meridian ML engineer and steals a personal access token for the code repository. With it, the attacker edits the pipeline definition for the customer-service assistant in a small commit titled "speed up CI": the safety evaluation stage now runs with continue-on-error: true, and a new step downloads a "tokenizer update" from an external URL.

The pipeline uses a long-lived cloud access key stored as a pipeline secret. Its build runners are long-lived virtual machines shared by all projects. The commit is merged by the same engineer's account. The next release ships with a weakened safety configuration and a modified tokenizer. Nobody notices for nine days.

By the end of this section you will be able to list every pipeline control that should have stopped this, and redesign the pipeline so that a stolen developer token is not enough to ship a model.

Threats along the pipeline

Figure 4.9 shows a secure MLOps pipeline, the threat at each stage, and the gates before promotion.

A secure MLOps pipeline with the threat at each stage and the gates before promotion
Figure 4.9 A secure MLOps pipeline with the threat at each stage and the gates before promotion
Stage Threat In the opening case
Source (code, prompts, configs) Malicious commit, secret leak Pipeline definition edited with a stolen token
Ingest Poisoned data —
Train / fine-tune Compromised job, bad dependencies External "tokenizer update" downloaded at build time
Evaluate Rigged or skipped metrics Safety evaluation set to continue on error
Package Unsafe format, vulnerable image —
Registry Unauthorized promotion Same account authored and merged
Deploy Unapproved model or config Weakened safety configuration shipped

Hardening CI/CD for models

  • Ephemeral runners: build and training jobs run on fresh, short-lived machines that are destroyed afterwards, so an attacker cannot persist between runs or read another project's secrets.
  • Federated credentials: pipelines obtain short-lived cloud credentials through identity federation (for example OpenID Connect from the CI system to the cloud), bound to the specific repository, branch and workflow, instead of storing long-lived keys as pipeline secrets.
  • Pinned inputs: base images, libraries, datasets, base models and tokenizers are pinned by version and hash, so a build cannot silently pull a changed or malicious component. The LiteLLM compromise in section 3.7 is exactly what pinning defends against.
  • No arbitrary downloads: builds fetch only from the internal mirror (section 3.7), with egress blocked otherwise.
  • Minimal, signed containers: small images with only what is needed, scanned and signed, with admission control that refuses unsigned images.
  • Safe model packaging: safe formats, scanning and signing (section 3.5).
  • Protect the pipeline definition itself: changes to pipeline files and gate configuration need review by someone other than the author, through branch protection and code-owner rules.

Protecting data ingestion and transformation

Organize data processing into zones of increasing trust, as Figure 4.10 shows.

Raw, validated and curated data zones with controls at each transition
Figure 4.10 Raw, validated and curated data zones with controls at each transition
  • Raw: data as received, with integrity checks and hashing on arrival.
  • Validated: checked, cleaned and scanned for personal data and anomalies.
  • Curated: approved for training, by the data owner, with a signed manifest (section 3.3).

Only the pipeline, not individuals, can write to the curated zone, and every promotion between zones is logged.

Lineage for fine-tuning

Fine-tuning jobs should capture lineage automatically: the exact base model version, dataset versions and hashes, hyperparameters, code version, who launched the job and when (section 3.7). Manual records are incomplete exactly when they matter. Lineage then answers four questions fast:

Question Situation
Which models used this data? Incident response: a source turns out to be poisoned
Which models learned from this person's records? An erasure request
Does this model inherit restrictive terms? Licensing from a base model or dataset
What was the last version built from known-good inputs? Rollback

Validation checkpoints and enforceable gates

Place checks along the pipeline: data validation before training; model evaluation (performance, fairness, robustness, safety and security) after training; artifact scanning before registration; and policy checks before deployment. A gate only protects if it is enforceable. Figure 4.11 shows how.

Enforceable gates: each stage signs an attestation, and the next stage verifies it before proceeding
Figure 4.11 Enforceable gates: each stage signs an attestation, and the next stage verifies it before proceeding
  • Attestations: each stage produces a signed statement of what it did, on which inputs and with what result (for example SLSA provenance [22]). The next stage verifies the attestations before proceeding, so a step cannot be skipped or faked. In the opening case, a skipped or failed safety evaluation would produce no valid "safety passed" attestation, and deployment would refuse to proceed.
  • Policy as code: deployment rules ("only signed models with a passing safety evaluation attestation, and committee approval for the high-risk tier") are written as machine-checked policies, evaluated by the deployment system, not as documents.
  • Separation of duties: the person who trains a model cannot approve it alone, and no single person can both change the pipeline and push a release without independent review.

Worked example: would the attack still work?

Attacker's step Redesigned pipeline
Uses stolen personal access token Token is short-lived and requires phishing-resistant MFA to issue (section 4.5); ideally the theft fails
Edits pipeline definition Branch protection: pipeline files need approval from a code owner other than the author
Sets safety eval to continue on error Deployment policy requires a signed "safety passed" attestation; a failed or skipped stage produces none
Downloads external "tokenizer update" Runner egress blocked; tokenizer pinned by hash from the internal mirror
Uses pipeline's long-lived cloud key No long-lived key exists; federated credentials are bound to the protected branch and workflow
Persists on shared runner Ephemeral runners destroyed after each job

Every step now fails independently. That is defense in depth applied to a pipeline.

Exam lens. "A required check was skipped" or "an unapproved model reached production" points to gates that were documentation rather than enforcement. The best answers are signed attestations verified by the next stage, policy as code, and separation of duties on both releases and pipeline changes.

Designing for forgetting

Sooner or later someone will need data removed from a model: a privacy erasure, withdrawn consent, a licensing dispute or poisoned records. Build unlearning into the pipeline from the start (sections 2.4 and 3.2):

  1. Lineage identifies the affected models.
  2. A deletion workflow removes the data from all datasets, indexes and caches.
  3. A suitable method is chosen: retraining from scratch without the data (most reliable, most costly); retraining from a checkpoint taken before the data was added; approximate machine unlearning; or retiring the model.
  4. Verification checks that the model no longer reproduces or reveals the data, for example with extraction and membership inference tests (section 3.3).

Designs that make forgetting cheaper: keep personal data out of training where possible (use retrieval, where deletion is simple); train on shards so only part needs retraining; and keep regular checkpoints with lineage showing which data each one includes.

The opening case, resolved

The controls that should have stopped the attack:

  • Identity: phishing-resistant MFA and short-lived tokens for developers.
  • Pipeline definition: branch protection with an independent code-owner review for pipeline files.
  • Gates: signed attestations for each stage and a deployment policy requiring a "safety passed" attestation, so a skipped evaluation blocks release.
  • Inputs: pinned tokenizer and dependencies from the internal mirror; egress blocked from runners.
  • Credentials: federated, short-lived cloud credentials bound to the protected workflow, replacing the long-lived key.
  • Runners: ephemeral, per-job.
  • Lineage and release: automatic lineage, so the bad release can be traced and rolled back to the last version built from known-good inputs, with canary metrics (section 3.5) to catch it in hours, not days.

Check your understanding

Try each before reading the reasoning.

1. A training pipeline stores a cloud access key as a CI secret so that jobs can read training data. What is the better design?

Reasoning: workload identity federation (for example OIDC from the CI system to the cloud) issuing short-lived credentials bound to the specific repository, branch and workflow. No long-lived key exists to steal.

2. A data scientist can write directly to the "approved for training" data bucket. Why is this a problem, and what is the fix?

Reasoning: it bypasses validation and approval, so poisoned or unapproved data can reach training. Use data zones: only the pipeline writes to the curated zone, after validation and data-owner approval, with every promotion logged.

3. Why is a "pass" result in a pipeline log not enough to prove that a model passed its safety evaluation?

Reasoning: logs can be edited or stages skipped. A signed attestation from the evaluation stage, bound to the exact model artifact hash and verified by the deployment stage, proves the check ran on that artifact and passed.

Key takeaways

  • In LLMOps, prompts, indexes, guardrail configurations and tool definitions are release artifacts too. Whoever controls the pipeline controls production.
  • Harden CI/CD with ephemeral runners, federated short-lived credentials, pinned inputs from an internal mirror, signed minimal containers and protected pipeline definitions.
  • Ingest data through raw, validated and curated zones; only the pipeline writes to curated.
  • Capture lineage automatically for response, erasure, licensing and rollback.
  • Make gates enforceable: signed attestations verified by the next stage, policy as code, and separation of duties for releases and pipeline changes.
  • Design for forgetting with lineage, deletion workflows, checkpoints and shards, and verify that forgetting worked.