Distribution Shift in Production Systems

Many teams follow the same routine. They build an evaluation set once, run the model against it, and record the resulting score. From then on they treat that score as a reliable signal for as long as the model is in use. The belief behind this habit is that an evaluation set, once built, keeps representing the data the model meets in production.

This article examines why that belief is wrong. The source material names the failure directly: an evaluation set is a living artifact, and it decays if it is not maintained. The source calls this eval-set rot. Before reading on, consider the question the source's framing raises. The set was accurate when it was built, so the decay cannot come from a flaw in its original construction. It must come from change in the world the model operates in.

The key word is relationship. A suite is not a fixed exam that measures the model in isolation. It is a stand-in for production. When production changes and the suite does not, the stand-in gets worse, even though the suite itself has not been altered at all. The score it produces is honest about the suite, but the suite is no longer an honest picture of production.

Once you see the evaluation set as something that must change in relation to production, rather than a finished artifact, the rest of the topic follows. The remaining question is how exactly the suite pulls away from production. The source identifies two mechanisms.

The source identifies two distinct mechanisms by which an evaluation set stops representing production. Both can be at work in the same suite at once, and each pulls the set away from production in a different direction. The first mechanism is that new edge cases appear in production and are never added to the suite. The second is that old cases stop reflecting what customers care about and are never retired from the suite.

In the first direction, production gains cases the suite lacks, so the suite is blind to them. In the second, the suite keeps cases that production no longer cares about, so a large share of the grading effort measures things customers have stopped caring about. The two directions produce the same symptom from the team's point of view: a score that is computed correctly but describes a world that has moved on.

The most troubling part of the source's description is that the team's confidence can rise while the customer's experience gets worse. This is not a contradiction. The team is measuring the model against the evaluation set, and the evaluation set is drifting away from production. Trace the two views side by side over successive periods.

At the first point the two views agree, because the suite was built from the production data of that time. As production moves and the suite is left alone, the two lines separate. The team's score stays flat and healthy because the model still handles the old cases well. The customer's experience drops because the model meets inputs the suite never included. The separation begins quietly and is invisible from inside the scoring pipeline.

The source describes this as the team grading itself against a suite that no longer represents the production distribution. The grade is honest about the suite, but the suite is no longer honest about production. That gap is why misplaced confidence is the central danger of eval-set rot.

The reasoning in this illustration matters more than its details. The product feature and the policy change are invented to show the two mechanisms in a concrete setting. What carries over to any real system is the pattern: a suite can keep producing a healthy number while its relationship to production erodes.

The source gives the remedy in one sentence: treat the evaluation suite as a living artifact with explicit ownership and a regular review cadence, adding and retiring cases as production changes. Each part of that phrase does work. Explicit ownership means a specific person or group is responsible for the suite, so updates do not depend on someone happening to notice a problem. A regular review cadence means the suite is checked on a schedule rather than only when something goes wrong.

The loop above is an illustrative sequence of the steps named in the source: review, add, retire, and re-baseline. The source does not prescribe this exact order or specify how often the cycle should run. What it does establish is that the suite should be reviewed on a regular schedule and that cases should be added and retired as production changes, under a named owner.