3.7 AI data and model supply chain risk
In this lesson you will:
- Assess a dataset from outside before it is used, against six questions.
- Use hash pinning and internal mirrors so that nothing changes underneath you.
- Recognize the supply-chain tricks used against model hubs and package registries.
- Verify models and packages with signatures and provenance attestations.
- Manage what you inherit when you build on someone else's foundation model.
Section 2.5 covered supplier governance: due diligence, contracts and AI bills of materials. This section looks at the technical risks in the artifacts themselves (datasets, models and packages) that flow in from outside [9][13].
Opening case: three weeks in October
Week 1. A Meridian developer building an internal document assistant adds a popular open-source LLM gateway library to her project by typing its name into the package installer. The library is widely used, and she pins nothing. A few days later, India's CERT-In issues an advisory that a release of the widely used LiteLLM Python package and its container images had been compromised with code that stole cloud credentials, SSH keys, CI/CD secrets and LLM provider API keys [33].
Week 2. Another team's training script loads a sentiment model by its hub path,
org-name/finance-sentiment. The original organization deleted its account months ago. Someone else has since registered the same organization name and uploaded a new model to the same path.Week 3. The procurement team asks whether Meridian can rely on its chosen foundation model provider's safety testing, "since they are the experts".
By the end of this section you will be able to say what Meridian should do in each week, and which controls would have made weeks 1 and 2 non-events.
Third-party, open and public datasets
Before a dataset from outside is used, assess it against six questions:
| Question | What to check | Why it matters |
|---|---|---|
| Poisoning | Has it been, or could it be, manipulated? | Web-scale datasets that point to URLs can change after publication [26] |
| Rights | Does the license allow this use? Were the people in it asked? | Legal exposure and takedown requests (section 2.4) |
| Personal data | Does it contain identifiable people, and on what basis? | Privacy obligations follow the data (section 3.2) |
| Harmful content | Illegal, abusive or toxic material? | The model learns what it sees |
| Bias | Whose data is over- or under-represented? | Unfair outcomes (section 2.2) |
| Provenance | Who created it, when and how, and can that be verified? | Without provenance, none of the other answers can be trusted |
Two controls do most of the work, and they apply equally to models and packages:
- Hash pinning records the cryptographic hash of the exact version that was assessed, and refuses any file that does not match. A later change, malicious or innocent, is detected.
- An internal mirror stores the approved copy in the organization's own repository. Pipelines pull only from the mirror, never directly from the internet.
Figure 3.24 shows an intake pipeline for external artifacts that combines them.
Verifying models and packages
Models and software come from public hubs and package registries, where attackers use familiar supply-chain tricks:
- Typosquatting: publishing a malicious model or package whose name is one character away from a popular one (
litellmversuslite11m). - Namespace reuse: when an author or organization deletes its account on a model hub, an attacker re-registers the same name and uploads a malicious model under the trusted path that existing code still references. This is week 2.
- Dependency confusion: publishing a public package with the same name as an organization's internal package, so build tools fetch the attacker's version.
- Compromised releases: a legitimate project's release process is hijacked, so the real package name delivers malicious code. This is week 1. In its October 2026 advisory, CERT-In advised identifying exposure from lockfiles and image digests, rebuilding from verified images rather than cleaning in place, and rotating every exposed secret [33].
Worked example: week 1, the compromised release
| Question | How Meridian answers it | Control that makes it fast |
|---|---|---|
| Are we affected? | Search all lockfiles and container image digests for the affected versions | Pinned versions and an AIBOM per system (section 2.5) |
| Where did it run? | Developer laptop, CI runner, and one staging container | Build and deployment logs |
| What could it steal? | Cloud credentials, SSH keys, CI/CD secrets and LLM API keys present in those environments | Secrets inventory |
| Response | Isolate; rebuild from known-good images (not clean in place); rotate every secret that was present; review provider and cloud logs for misuse | Incident playbook (Module 5) |
| Prevention next time | Install only from the internal mirror, with exact versions pinned by hash, and a delay before new upstream releases are approved | Internal registry with approval step |
If Meridian had pulled packages only from an internal mirror with a short quarantine on new upstream releases, the compromised version would very likely never have reached the developer.
Worked example: week 2, namespace reuse
The training script trusted a name, not an artifact. The fix is to reference models by an immutable identifier: the exact revision and hash of the approved version, pulled from the internal mirror. Then a new upload to the same path is simply a different artifact that has not been approved, and nothing loads it.
Defenses in summary
- Download only from verified publishers, and pin exact versions by hash.
- Verify signatures and provenance attestations where available. SLSA provenance describes how and where an artifact was built; Sigstore signatures prove who signed it [28][29].
- Scan model files for embedded code and prefer safe formats such as safetensors (section 3.5).
- Use an internal registry or mirror with an approval step and a quarantine period for new upstream versions.
- Reserve your internal package names in public registries to block dependency confusion.
- Maintain the AIBOM, so exposure can be found in minutes, not days.
Risks inherited from foundation models
Building on a pre-trained or foundation model means inheriting what you cannot see: its training data (with any poisoning, bias, copyrighted or personal content), its safety tuning and its weaknesses. Fine-tuning adds your own layer on top, and lineage must connect the two, as Figure 3.25 shows.
Manage inherited risk across the model's whole life:
| Stage | What to do |
|---|---|
| Selection | Prefer providers with transparency about training data, published evaluations and security practices; check license terms along the pedigree |
| Evaluation | Test the model yourself for your use: performance, bias, jailbreak resistance and data leakage. Public benchmarks are not your use case |
| Safe operation | Wrap the model in your own guardrails, access controls and monitoring. Do not assume the provider's safety tuning is enough |
| Re-testing | Repeat evaluation whenever the base model, adapter or provider version changes, since behavior can shift (section 1.3) |
| Exit planning | Know how you would switch models if the provider withdraws it, changes terms, suffers a breach, or the model is found to be compromised. Keep prompts, evaluation sets and integration layers portable |
Fine-tuning has a specific inherited risk worth remembering: research and practice have shown that fine-tuning, even on harmless data, can weaken a base model's safety behavior. A fine-tuned model is a new model and needs its own safety evaluation.
Exam lens. "Can we rely on the provider's testing?" is answered by shared responsibility: the provider's testing covers the base model in general; you must evaluate it for your use, wrap it in your own controls, and re-test on every change.
The opening case, resolved
- Week 1 (compromised LiteLLM release): find exposure through lockfiles, image digests and AIBOMs; isolate; rebuild from verified images rather than cleaning in place; rotate every secret present in affected environments; check logs for misuse. Prevent recurrence with an internal mirror, pinned hashes and a quarantine period for new releases.
- Week 2 (namespace reuse): stop the training job; check whether the new model was loaded and in what format (if pickle, treat the machine as compromised); reference models by pinned revision and hash from the internal mirror, never by mutable hub path.
- Week 3 (relying on the provider): no. The provider's testing is necessary but not sufficient. Meridian must evaluate the model for its own uses, add its own guardrails and monitoring, re-test on every model change, and plan for exit.
Check your understanding
Try each before reading the reasoning.
1. A build system fetches an internal package called meridian-risk-utils. An attacker publishes a higher-versioned public package with the same name, and the build picks it up. What attack is this, and what prevents it?
Reasoning: dependency confusion. Prevent it by configuring builds to resolve internal names only from the internal registry, reserving internal names on public registries, and pinning versions by hash.
2. A team wants to verify that a model artifact was built by the claimed pipeline from the claimed source. What should they check?
Reasoning: provenance attestations (for example SLSA provenance) describing how and where it was built, together with a signature (for example Sigstore) proving who signed it, and a hash match against the approved version.
3. After a vendor fine-tunes a base model on Meridian's harmless FAQ data, should the original safety evaluation of the base model still be relied on?
Reasoning: no. Fine-tuning can weaken safety behavior, even on harmless data. The fine-tuned model is a new model and needs its own safety and jailbreak evaluation before release.
Key takeaways
- Assess external datasets for poisoning, rights, personal data, harmful content, bias and provenance before use.
- Hash pinning and internal mirrors stop artifacts changing underneath you, for datasets, models and packages alike.
- Watch for typosquatting, namespace reuse, dependency confusion and compromised releases. Reference artifacts by hash, not by name.
- Verify signatures and provenance attestations, scan model files and prefer safe formats.
- After a compromised release, rebuild from verified images and rotate every exposed secret.
- Foundation model risk is inherited: evaluate for your use, add your own controls, re-test on every change (fine-tuning included), and plan your exit.