The Model Provenance Gap: Why You Cannot Prove Which Model Produced a Decision in Production

The Question Nobody Can Answer

Here is a question that sounds trivial and is not: for a specific decision your AI system made three weeks ago, which exact model produced it? Not the model family. Not the one your architecture diagram says you use. The precise weights, version, prompt template, and context that generated that particular output.

In most production AI systems, nobody can answer this. The team knows they run a GPT-class or a fine-tuned model. They can point at a config file that describes the intended setup. But between the decision and the description sits a stack of runtime machinery -- routers, fallbacks, hot-swaps, prompt assembly, retrieval -- that mutates what actually ran, request by request, in ways the logs do not faithfully capture. The model provenance gap is the distance between the decision you shipped and your ability to prove, after the fact, what produced it.

This is not a theoretical concern for a compliance slide. It is the question a regulator asks during an audit, the question a customer's lawyer asks in a dispute, and the question your own incident team asks at 2 a.m. when a system starts making decisions it should not. In each case, we-think-it-was-probably-the-primary-model is not an answer. It is an admission that you do not govern the system you deployed.

Why Provenance Erodes in Production

The gap does not open because teams are careless. It opens because every reliability and cost optimization you add to a production AI system actively degrades traceability, and nobody accounts for the tradeoff.

Model routing decouples intent from execution. The moment you route requests across models by cost, latency, or capability, the model that handled any given request is a runtime decision, not a static fact. If your logs record the router's intent rather than the router's actual choice, you have already lost provenance. This is the direct governance cost of hot-swap model routing in production, where failover happens without downtime -- and often without a durable record of which model actually served the request.

Fallback chains rewrite history silently. When the primary model times out and the request cascades to a secondary or a cheap local fallback, the output came from a different artifact than the one you intended. If that substitution is not stamped onto the decision record, your audit trail confidently attributes the output to a model that never saw the request. This is the provenance shadow of the fallback cascade problem, where failover chains behave differently than the system you tested.

Configuration drifts away from what you tested. The model serving traffic today may not be the one your evaluation certified, because weights, quantization, provider defaults, and inference parameters shift underneath you. This is configuration drift in AI systems, where your production model is not the one you tested -- and if you cannot pin the exact configuration to each decision, you cannot even say which version's behavior you are looking at.

Prompt assembly is unversioned and scattered. The model is only half the artifact. The prompt -- system message, injected context, retrieved chunks, tool definitions -- is the other half, and it is often assembled at runtime from a dozen scattered code paths. The output is a function of model plus prompt, but most teams version neither together. This is exactly the ungovernability described in prompt assembly sprawl, where scattered prompt construction becomes impossible to reconstruct at scale.

Stack these four and the result is stark: the decision your system made is the product of a model you cannot pin, a prompt you did not version, a route you did not record, and a fallback you did not stamp. Provenance did not fail at one point. It leaked out of every seam.

Why Having Logs Is Not Having Provenance

Most teams, told they have a provenance gap, point at their logging. This is the core misunderstanding. Logs are typically records of what the code intended to do -- they emit the configured model name, the template id, the router's target. They are written by the same code whose behavior you are trying to audit, and they capture intent, not execution.

Provenance is different. Provenance is an immutable, decision-scoped record of what actually produced a specific output: the resolved model identity and version, the fully materialized prompt, the retrieved context with its source identifiers, the inference parameters, and the routing and fallback path actually taken. It has to be captured at the moment of execution, bound to the decision, and tamper-evident after the fact. A log that claims the request was routed to primary when it actually fell through to a fallback is worse than no log -- it is confidently wrong, and it will survive an audit right up until someone checks it against reality.

This is the same distinction that separates real explainability from theater. It is why AI audit trails are not optional, and why explainability has to be engineered into the system rather than bolted on as a post-hoc reconstruction that nobody can actually verify.

What the Gap Costs When It Matters

Regulatory exposure. Under emerging AI governance regimes, which-model-made-this-decision-and-on-what-basis is not optional documentation -- it is the audit. A firm that cannot attribute a decision to a specific, versioned artifact cannot demonstrate compliance, cannot scope the blast radius of a bad model, and cannot prove a fix actually removed the offending version from production. For smaller organizations racing to formalize this, it is the missing foundation under any AI governance framework, which is meaningless if you cannot tie decisions to the models that made them.

Un-scopable incidents. When a model starts producing harmful outputs, your first job is to determine which decisions it touched. Without provenance, you cannot draw that boundary. You either over-recall -- treating every recent decision as suspect and eroding trust -- or under-recall and miss affected cases. Both are governance failures, and both flow directly from the inability to attribute.

Undefendable disputes. When a customer challenges a decision, being unable to reconstruct exactly what your system did is not a neutral answer. In a regulated or contractual context, it shifts the presumption against you. The provenance gap converts a defensible decision into an indefensible one purely because you cannot show your work.

Engineering Provenance Into the System

Provenance is not a feature you add later. It is an architectural property you either design in or permanently lack. Retrofitting it onto a system that was not built for it is close to impossible, because the information you needed was destroyed at execution time.

Stamp the resolved artifact, not the intended one. Every decision record must capture the model identity and version that actually served the request -- resolved after routing and fallback, not the router's intent. If a request fell through to a secondary, the record says secondary. This closes the routing and fallback leaks at their source.

Version the model and prompt as a single unit. The artifact that produces a decision is model plus fully materialized prompt plus retrieval context. Bind them into one immutable, hashable decision envelope. If you cannot reproduce the exact input from the record, you do not have provenance -- you have a guess.

Make the record immutable and decision-scoped. Provenance that can be edited is not provenance. Write decision records to an append-only, tamper-evident store, keyed to the individual decision, so that months later you can retrieve the exact envelope and verify it was not altered. This is the same discipline behind serious observability for AI systems, which is nothing like traditional APM because the thing you need to reconstruct is a probabilistic decision, not a stack trace.

Enforce provenance as a hard boundary, not a best effort. A request that cannot be provenance-stamped should fail closed, not proceed unrecorded. This belongs in the control plane, alongside the other hard limits that keep autonomous systems governable -- the same philosophy as deterministic control planes for agentic AI, where the orchestrator enforces boundaries the model cannot cross. Provenance capture is one of those non-negotiable boundaries.

Treat provenance as data that consumers depend on. Downstream compliance, incident response, and analytics all consume the provenance record, which means its schema deserves the same rigor as any other critical interface. Apply data contracts to your provenance records, so a change to how you stamp decisions cannot silently break the audit trail that depends on it.

The Uncomfortable Bottom Line

Most teams shipping AI into production have optimized for reliability and cost with routing, fallbacks, hot-swaps, and dynamic prompt assembly -- and in doing so have quietly traded away their ability to answer the single most important governance question: what actually made this decision. They will not discover the gap until an auditor, a court, or an incident forces the question, and by then the information is gone, because it was never captured at execution time.

Provenance is the price of running AI you can be held accountable for. If you cannot attribute a decision to a specific, versioned, reproducible artifact, you are not operating a governed system -- you are operating a plausible-deniability machine, and eventually someone will ask you to prove otherwise.

Close the Gap Before Someone Forces the Question

If your team cannot currently answer which-exact-model-produced-this-decision for an arbitrary production output, that is not a documentation gap -- it is an architectural one, and it compounds every day you ship without fixing it. Book a working session with Bigyan Analytics and we will map where provenance leaks out of your current pipeline and what it takes to make every decision attributable.