Model Supply Chain Needs Provenance Checks

AI model supply chain security means verifying where a model came from, who built it, and what is inside it before you let it run in production. A model file is not a neutral blob of numbers. It is a software artifact, pulled from a hub, wrapped in a container, and often loaded with a single line of code that trusts the file completely. If that trust is misplaced, the model can carry bad weights, a hidden license problem, or executable code disguised as data.

The pattern is not new. Software teams learned this lesson with open source packages, and the response was frameworks like SLSA, which defines levels of build integrity so a team can prove a package was built the way it claims to have been built. Model files deserve the same discipline, because a compromised model can leak data, produce unsafe output, or execute code the moment it loads, just like a compromised package can.

On this page

• Where models actually come from
• An unverified model is an unverified package
• Provenance and signing for model artifacts
• Training data lineage and license checks
• Scanning model files for malicious code
• Pin versions instead of pulling latest
• Keep a bill of materials for models
• FAQ

Where models actually come from

Few teams train a model from scratch. Most start with pretrained weights published by a lab, then fine-tune on their own data, then package the result in a container for deployment. Each of those steps adds a new source to track: the base model, the fine-tuning dataset, the training script, and the runtime image that serves it.

Third-party hubs make this easy and that ease hides the risk. Anyone can upload a model file to a public hub under a name that sounds official. A team downloading “the” version of a popular model may actually be pulling a re-upload with altered weights or a different license than the original.

• Pretrained base weights from a research lab or vendor
• Fine-tuned checkpoints built on top of that base
• Container images that bundle the model with its serving code

An unverified model is an unverified package

Security teams already know not to run a random script from the internet without checking it. A model file gets less scrutiny even though loading it can trigger code execution, download remote resources, or shape every decision an application makes downstream. That gap in scrutiny is the actual risk.

The comparison to package management is not a metaphor, it is the same problem in a different file format. A poisoned dependency and a poisoned model both enter through a trusted-looking download, both run with the permissions of the process that loads them, and both can sit undetected until something goes wrong in production.

• Unknown origin means unknown intent behind the weights
• Model loaders often execute code as part of loading, not just data
• A single compromised model can affect every application built on it

Provenance and signing for model artifacts

Provenance answers a simple question: can you prove how this model was built and by whom. The SLSA framework was designed for software builds, but its core idea, a verifiable record connecting source code to the final artifact, applies directly to model training pipelines and their output files.

Signing closes the loop. A tool like Sigstore lets a publisher sign an artifact and lets anyone verify that signature without managing long-lived keys themselves. Applying that same signing step to model files means a team can check, before deployment, that the file matches what the publisher actually released.

• Provenance records link a model file back to its training run
• Signing proves the file has not been altered after publishing
• Verification should happen automatically, not by manual trust

Training data lineage and license checks

Before a model goes into production, someone should be able to answer what data trained it and under what terms. Training data lineage is not a formality. A model trained on data with restrictive licensing can create legal exposure the moment it ships in a commercial product.

This check matters as much for fine-tunes as for base models. A team fine-tuning on internal data still inherits the license terms of the base model, and a base model with unclear provenance makes that inheritance impossible to verify. Skipping this step trades a small amount of upfront work for an open-ended liability later.

• Confirm the base model license permits your intended use
• Document the datasets used in any fine-tuning step
• Flag models with missing or unclear lineage before approval

Scanning model files for malicious code

Some model file formats can carry executable code alongside the weights. Python’s pickle format, used historically to save and load many machine learning models, deserializes objects in a way that can run arbitrary code during loading. This is a documented and well known issue, not a theoretical one, and it is why newer formats designed for safety avoid this behavior entirely.

A scanning step before deployment catches what a quick glance cannot. Treat every downloaded model file the way a security team treats an unfamiliar binary: assume it needs inspection until proven otherwise, and prefer formats that store only tensors and no executable logic.

• Scan pickle-based files for embedded code before loading them
• Prefer weight formats that cannot execute code on load
• Run untrusted models in an isolated environment first

Operating rule: never load a model file directly into a production environment before it has passed a provenance check, a license check, and a code scan. Treat the model exactly like you would treat an unreviewed software dependency.

Pin versions instead of pulling latest

Auto-pulling the latest version of a model sounds convenient and creates an invisible risk. A hub maintainer can replace a model’s weights under the same name and tag, and an application that always fetches “latest” will silently start running a different model than the one it was tested against.

Pinning a specific version, identified by a fixed hash or tag, keeps behavior predictable and keeps security reviews meaningful. A review of a model only holds if the model that gets deployed is the exact one that was reviewed, not whatever happens to be published under that name later.

• Pin models by a fixed identifier, not a mutable tag like latest
• Re-run the full review process before adopting a new version
• Track which pinned version is running in each environment

Keep a bill of materials for models

A software bill of materials lists every component in an application so a team can respond quickly when a vulnerability is disclosed. Models deserve the same inventory. When a base model or a widely used checkpoint turns out to have a problem, a team needs to know instantly which applications use it.

Building this inventory does not require new tooling from scratch. It requires discipline: recording the model source, its version, its license, its provenance status, and where it is deployed, then keeping that record current as models change. Frameworks like the NIST AI Risk Management Framework point toward this kind of structured accountability across the model lifecycle.

• Record source, version, license, and provenance status per model
• Map each model to the applications and environments that use it
• Review the inventory on a fixed schedule, not only after an incident

FAQ

What is AI model supply chain security in simple terms?

It is the practice of verifying where a model comes from, who built it, and what it contains before deploying it, the same way software teams verify a package before adding it to a build.

Why is a model file treated as a security risk?

Because loading a model can execute code, not just read numbers. A file from an unverified source can carry altered weights or hidden executable logic that runs the moment it loads.

What does SLSA have to do with AI models?

SLSA defines levels of build integrity for software artifacts. The same principle, a verifiable link between source and final output, applies to model training pipelines and the weight files they produce.

Why is the pickle format a concern for model files?

Pickle deserializes objects in a way that can run arbitrary code during loading. This is a known and documented behavior, which is why safer formats that store only tensors are preferred for model distribution.

Should teams always use the latest version of a model?

No. Auto-pulling the latest tag means a model can change under an application without notice. Pinning a fixed version keeps deployments predictable and keeps prior reviews valid.

What belongs in a model bill of materials?

The model’s source, version, license, provenance status, and every application or environment where it is deployed, kept current as models are added, updated, or retired.

Does checking license terms really matter for a fine-tuned model?

Yes. A fine-tuned model still carries the license terms of its base model, so unclear provenance on the base model makes it impossible to confirm the fine-tune is safe to use commercially.

Related reading

• Training Data Poisoning Needs Source Control
• AI Security Needs Its Own Threat Model

Conclusion

Treat every model file as a piece of software that needs verification, not a static file that can be trusted on sight. Check its provenance, confirm its license and training data lineage, scan it for embedded code, pin the version you deploy, and keep an inventory of what is running where. These are not extra steps bolted onto machine learning work. They are the same supply chain discipline that software teams already apply, adapted to a new kind of artifact.

Sources

• SLSA v1.0 Specification
• Sigstore Documentation
• NIST AI Risk Management Framework