The Prompt Version Skew Problem: Why Your A/B Test Compares Two Prompts That Were Never Actually Live Together

The Experiment That Measured Everything Except the Prompt

You did it the disciplined way. Two prompt variants, a routing layer to split traffic, a primary metric, a dashboard. After two weeks, prompt B wins by a comfortable margin on resolution rate. You promote B, archive A, write the eng-notes, and move on. Clean science.

Except it wasn't. When you pull the raw event logs, an uncomfortable pattern emerges: prompt A served the bulk of its traffic in the first four days of the experiment, and prompt B served most of its traffic in the second week. In between, the model provider silently rolled a point release of the underlying endpoint, your retrieval index got a scheduled re-embedding, and a marketing push shifted the inbound traffic mix from power users toward first-timers. Prompt A and prompt B were never truly live under the same conditions. You did not run an A/B test. You ran two sequential observations of a moving system and attributed the difference to the one variable you happened to be watching.

This is prompt version skew: the confound that appears when the thing you are comparing is entangled with time, and everything else about the system is drifting underneath your experiment. In classic web A/B testing, randomization at the request level plus a stable backend makes this a non-issue. In production LLM systems, the backend is not stable, the prompt is not the only moving part, and "version" is a fiction unless you engineer it to be real.

Why LLM Systems Are Uniquely Prone to Skew

Three properties of production AI systems conspire to make prompt experiments lie to you.

The model under the prompt is not a fixed function. Hosted endpoints get updated, quantized, re-routed, and load-balanced across hardware without a version bump you control. The same prompt against the "same" model on Tuesday and the following Monday can produce measurably different outputs. If your experiment does not pin and record the exact model build serving each request, you cannot separate prompt effect from model drift. This is the same accountability hole described in the model provenance gap, where you cannot prove which model produced a decision in production -- and an A/B test without provenance is just a rumor with a p-value.

The prompt is rarely a single string. In any real system the "prompt" is assembled at request time from a system template, retrieved context, tool schemas, few-shot examples, and user input. Change the retrieval index mid-experiment and prompt B is effectively a different prompt than it was on day one -- even though the template you are testing never changed. This is prompt assembly sprawl, the pattern that makes AI systems ungovernable: if you cannot reconstruct the exact assembled prompt for a given request, you cannot attribute the outcome to your variant.

The whole system drifts even when nothing is deployed. Config changes, dependency updates, and index refreshes accumulate silently between the start and end of an experiment. That slow, unversioned drift is configuration drift in AI systems, and a long-running A/B test is precisely the window in which it does the most damage -- because it correlates with time, and so does your traffic allocation.

The Statistics Break in a Specific Way

Skew is not just noise; it is bias with a direction. If prompt B happened to run more heavily during a period of easier traffic, cleaner context, or a better model build, the entire measured lift can be an artifact of when B ran rather than what B said. Sequential or ramped rollouts -- start A at 90/10, ramp B up over the week -- are the worst offenders, because they guarantee that variant and time are correlated by construction. You have built the confound into the experiment design itself.

Even properly randomized 50/50 splits are not immune. If a model endpoint updates halfway through, both variants are affected, but not necessarily equally: the update may interact with B's longer context window or A's specific formatting in ways that shift the delta. The comparison is only valid within a regime, and nobody told you when the regime changed.

How to Run Prompt Experiments That Actually Compare Prompts

The fix is to make "version" a real, enforced object and to collapse the time dimension out of your comparison.

Pin and stamp every dependency per request. Log the exact model build, the retrieval index version, the assembled prompt hash, the tool schema version, and the config snapshot alongside every experiment event. If you cannot reconstruct the full input that produced an output, the datapoint is unusable. Treat these as enforced boundaries, the way data contracts for AI pipelines treat schema at every hop -- a versioned interface, not a convention.

Randomize at the request level and analyze within regime. Never ramp; split concurrently so both variants experience the same drift at the same time. When a dependency changes mid-experiment, segment the analysis at that boundary and check that your delta holds in each regime. A result that only appears in one regime is a regime effect, not a prompt effect.

Route deterministically so the experiment is reproducible. The allocation, the version pinning, and the assembly should run through a controlled layer rather than being scattered across services. This is the case for deterministic control planes in agentic AI: if you cannot replay the experiment with the same inputs and get the same routing, you cannot trust the outcome.

Instrument the comparison as a first-class evaluated artifact. Do not judge prompt changes on a single production metric watched through a drifting window. Anchor them to a held-out eval set that runs against both variants under identical, pinned conditions -- the discipline of eval-driven development for testing AI systems. Production A/B tells you what happened; a controlled eval tells you why, and the two together are what let you attribute a win to the prompt rather than to the calendar.

The Takeaway

Most prompt A/B tests in production are not measuring prompts. They are measuring the entangled sum of a drifting model, a shifting index, a changing traffic mix, and -- somewhere in the noise -- the prompt you thought you were testing. Prompt version skew is what happens when "version" is a label you write in a doc instead of an object your system enforces. Pin every dependency, randomize concurrently, analyze within regime, and anchor to controlled evals. Otherwise you are not shipping the better prompt. You are shipping the one that got lucky with its timing.

---

Bigyan Analytics builds the governance and evaluation infrastructure that makes AI experiments trustworthy -- versioned prompts, pinned models, and reproducible control planes. Book a working session to pressure-test your AI experimentation stack.