Evaluation Set Design for AI Models
An evaluation set is a living artifact that decays over time if it is not maintained, a failure the source calls eval-set rot.
A Score That Keeps Looking Healthy
Many teams follow the same pattern. They build an evaluation set once, run the model against it, and record the resulting score. From then on, they treat that score as a reliable signal for as long as the model stays in use. The habit rests on an unstated belief: that an evaluation set, once built, keeps representing the data the model meets in production.
This article examines why that belief is wrong. The source material describes the evaluation set as a living artifact that decays over time if it is not maintained. The source calls this failure eval-set rot. Once you see the set as something that must change in step with production, rather than a fixed exam, the rest of the topic follows.
The term is useful because it names a failure that is easy to miss. Nothing in the evaluation pipeline breaks. The scripts run, the score is produced, and the report looks normal. The rot lives in the relationship between the suite and production, and that relationship does not show up in the score alone.
Following a Suite Through Time
Trace what happens to one evaluation set as months pass. At the start, the suite is a fair picture of the traffic the model receives. Then production changes. New kinds of requests appear, and some older requests stop mattering to customers. The suite itself does not change unless someone changes it, so it keeps grading the model against a picture of production that is no longer current.
The diagram below shows the two tracks that run side by side. The team track is the score the team sees. The customer track is what real users experience. Both move forward in time, but nothing forces them to move together.
Read the diagram from left to right. The team track ends high because the team is still grading against a suite that matches the production of several months earlier. The customer track ends low because the production it reflects has moved on. Neither track is wrong by itself. The problem is the gap that opens between them.
Two Ways the Suite Drifts
The source identifies two separate mechanisms by which an evaluation set stops representing production. The first is that new edge cases appear in production and are never added to the suite. The second is that old cases stop reflecting what the customer cares about, and they are never retired. Both mechanisms can be active in the same suite at once, and each pulls the set away from production in a different direction.
The first mechanism is an omission. Production produces a case the suite has never seen, and nothing in the suite ever tests it. The second mechanism is an accumulation. The suite keeps cases that once mattered to customers but no longer do. Those cases still consume the score, so the score reflects performance on a set of needs that customers have stopped having.
Eval-set rot is the gradual decay of an evaluation set that is not maintained, as the set drifts away from the production distribution it was built to represent.
A Worked Drift Scenario
How a refund-policy change splits the two mechanisms
A team runs a support assistant that answers questions about returns. Their suite was built when the store only accepted returns within 30 days. Later, the store changes its policy to offer store credit for some returns, and it launches a new product category. Using the two drift mechanisms, explain how the suite drifts.
- 1Identify the new cases
Customers now ask about store credit and about the new product category. These are new edge cases that production sends, but the suite has no examples of them. This is mechanism one, new edge cases not added.
- 2Identify the stale cases
Suite cases that check only the old 30-day return rules still run and still score the model. If customers no longer ask those questions in the same way, those cases no longer reflect what customers care about. This is mechanism two, old cases not retired.
- 3Combine the effects
Both mechanisms act at once. The score can stay high because the model answers the old rules well, while the suite never tests the store-credit questions that customers now ask most.
The suite drifts in two directions at once. It is missing the new cases and still grading on cases that have lost relevance, so the score can look healthy while customers meet problems the suite never tests.
Why Confidence Rises While Experience Falls
The most troubling part of the source's description is the observation that the team's confidence goes up while the customer's experience goes down. This is not a contradiction. The team is measuring the model against the evaluation set, and the evaluation set is drifting away from production. If the model handles the old cases well and the team has not added the new ones, the score can keep looking healthy while real users run into problems the suite never tests.
The source describes this as the team grading itself against a suite that no longer represents the production distribution. The grade is honest about the suite, but the suite is no longer honest about production. The gap between those two facts is why misplaced confidence is the central danger of eval-set rot.
Running the Suite as a Living Artifact
The source gives the remedy in one sentence. Treat the evaluation suite as a living artifact with explicit ownership and a regular review cadence, adding and retiring cases as production changes. Each part of that phrase does work.
Explicit ownership means a specific person or group is responsible for the suite. Updates then do not depend on someone happening to notice a problem. A regular review cadence means the suite is checked on a schedule, rather than only when something goes wrong. Together, these turn the suite from a fixed exam into something that is kept current.
Read the loop as a repeating cycle. Each pass starts by looking at production, then adds what is missing and retires what has gone stale, then reruns the suite. The loop closes with the next scheduled review. The owner drives each step, which is what the source means by explicit ownership. The fixed schedule is what the source means by a regular review cadence.
Mistakes Teams Make With Evaluation Sets
Common mistake Treating the eval set as a fixed exam built once
Why it failsThe source describes the set as a living artifact that decays if it is not maintained. A set built once will drift as production changes.
Do this insteadGive the suite an owner and a review schedule, and update it as production changes.
Common mistake Adding cases only when a failure is reported
Why it failsThis makes the suite react to problems customers have already hit, rather than keeping it aligned with what production sends.
Do this insteadReview recent production traffic on a regular cadence and add new edge cases before they cause harm.
Common mistake Never retiring old cases
Why it failsCases that no longer reflect what customers care about still count toward the score, which keeps the reported result tied to needs customers have stopped having.
Do this insteadRemove cases that have stopped mattering, as part of the same review cycle that adds new ones.
Common mistake Reading a high score as proof the model is healthy
Why it failsThe source notes that confidence can rise while customer experience worsens, because the team grades against a suite that no longer matches production.
Do this insteadCheck whether the suite still represents production before trusting the score.
Practice: Name the Drift Mechanism
A team's suite was built for a chat assistant that handled only billing questions. The product now also handles shipping delays, and customers have stopped asking about the old paper-invoice billing flow. The team's score has stayed flat for six months. Which drift mechanism is at work for shipping delays, and which is at work for the paper-invoice cases? Explain why a flat score does not rule out drift.
Hints
- Ask whether the shipping-delay cases exist in the suite at all.
- Ask whether the paper-invoice cases still reflect what customers care about.
- Think about whether a flat score can be produced by a suite that no longer matches production.
What do you think happens?
Does a flat score over six months show that the suite still represents production?
Reveal answer
No, a flat score can hide drift because the suite may no longer match production
The score measures the model against the suite, not against production. If missing cases and stale cases are both present, the score can stay flat while the suite drifts away from what customers experience.
Key Takeaways
- Eval-set rot is the decay of an evaluation set that is not maintained, as it drifts away from the production distribution it was built to represent.
- The suite drifts through two mechanisms: new production edge cases are not added, and old cases that no longer matter to customers are not retired.
- A team's confidence can rise while the customer's experience worsens, because the team grades the model against a suite that no longer represents production.
- The remedy is to treat the evaluation suite as a living artifact with explicit ownership and a regular review cadence, adding and retiring cases as production changes.