We commit to develop the infrastructure that captures the lineage of our AI systems' operations, so historical steps can be reconstructed and reproduced, enabling Provenance by Reproducibility.
Diagnosing a production incident requires reproducing the conditions that caused it. A team deploying a model it did not train cannot reproduce the model, but they can and must be able to reproduce operations on their own system. Foundational pillars of AI provenance include pinned model versions, the prompts and configuration in place, the data that flowed through, and the provenance of any fine-tune, including base model, data and adapters. Hosted models are updated and deprecated by their providers, so an unpinned version changes the system underneath its operators and must be explicitly captured.
Reproducibility is also what makes auditability possible, not only lineage. For AI systems it is necessary to record not only decisions but also trajectory metadata required to reconstruct what specific steps were previously carried out. Retaining these records is now an operational capability that is foundational for production best practices for any AI systems in production.
01 — WHERE IT FAILS
Where it fails
A single missing trail can have a cascading impact on the provenance of the system, and the gap is usually discovered during the incident that needed it.
- Unpinned model versions that let providers change the system underneath its operators.
- Gaps in data governance across the system’s operations.
- Historical steps that cannot be reconstructed from what was recorded.
02 — PRACTICAL CONTROLS
Practical controls
Controls should make the team’s own system reproducible even when the model it builds on is not theirs.
- Pin versions for models, prompts and configuration, including the base model, data and adapters of any fine-tune.
- Make atomic steps deterministic through seed and input metadata.
- Record data provenance and lineage across operations, including the trajectory metadata needed to reconstruct past steps.