Regression Testing for Prompt Changes

Imagine a team that builds an evaluation set for an AI feature in its first week. The cases reflect what the team expects users to ask and what a good answer should look like. Every prompt change is then run against that set, and the team watches the score. For a while, the score tells them something useful. Then, a few months later, the feature has met real users, the questions have shifted, and the team understands its customer much better than it did at the start. The score still goes up and down, but it increasingly measures a product that no longer exists in practice.

This article is about that problem and its remedy. The central claim is that an evaluation set is a living artifact. It has to change as the team learns more about the customer's domain, and it should be maintained the way you maintain code, not finished once and filed away.

When a team first builds an evaluation set, the set usually reflects what the team knows at that moment. Its cases are drawn from early expectations about what users will ask and what a good answer looks like. Those expectations are rarely complete. As the feature meets real users, gaps in the original set become visible. A set that was accurate in the first week can quietly stop measuring what matters by the third month.

The reasoning is simple. An evaluation set is only useful if it tests for the things the customer cares about right now. Customer needs shift, new kinds of input appear, and some failure modes that once mattered stop occurring or stop being important to the business. If the set stays fixed, it keeps measuring an older version of the product, and a passing score tells you less and less.

The difference between the two approaches is easiest to see side by side. A static set is built once and then left alone, so its contents stay tied to the early expectations. A living set is revised as evidence arrives: new edge cases are added, obsolete cases are retired or down-weighted, and the set continues to describe present needs.

Notice that the living set is not defined by being bigger. Its value comes from the fact that each case still corresponds to something the team currently needs to check.

As the team learns more about the domain, new edge cases will surface. These are inputs the original set did not anticipate, or situations where the product behaves in a way the team had not considered. When one appears, the correct response is to add it to the evaluation set so that future changes are checked against it. The set should grow to cover what the team has learned.

How you record an edge case matters. Good sources of evidence include the following:

Each of these is evidence about a situation the evaluation set should be able to judge. An edge case is therefore not just a tricky input; it is a recorded situation with a clear reason for being in the set.

Evolution works in both directions. A case enters the set when new evidence reveals a need, it can stay central while it keeps testing something the customer cares about, and it can later be down-weighted or retired when it stops mattering. The diagram follows one case through these states.

Retiring an eval removes it from the set. Down-weighting keeps it in the set but reduces how much it counts in the overall picture. Down-weighting is useful when a case is no longer central but still has some historical value. The transition from active to down-weighted is therefore a judgment that the case still informs the team while no longer driving decisions.

The reason to remove or de-emphasize old cases is that a stale eval can distort decisions. If a large share of the set checks behavior that no longer matters, a team may hesitate to ship an improvement that helps current customers, because the improvement appears to hurt a case nobody cares about anymore. Keeping the set focused on present needs keeps the signal clean.

The following example is an illustrative sketch, not taken from the source. It uses a set of six case labels, A through F, to show how the contents change over three versions. Each version records the change that produced it, so you can trace what was added, retired, or down-weighted at every step.

Read the timeline carefully. The set never grew without bound and never shrank into irrelevance. Each version changed only the cases whose status had shifted, and the set at every version can be explained by the sequence of decisions that led to it.

The same principle can be pushed one step further. Every time a prompt is revised, the revision is usually a response to an observed problem. Someone saw an output that was wrong, unhelpful, or unsafe, and changed the prompt so the system would avoid it next time. That revision carries information. It encodes a lesson about what bad outputs look like for this product.

That lesson should not be lost once the prompt is fixed. It should become an eval. Instead of relying on memory or on a careful reviewer to remember why a prompt changed, the team records the failure the revision addressed as a check that future changes must pass. A data-quality assertion is one form such a check can take: a rule that outputs of a given kind must satisfy.

The revision history can also be mined more systematically. Rather than writing every assertion by hand, the team can treat the sequence of revisions and the failures behind them as material from which candidate assertions are generated. A person still reviews the candidates to decide which are sound. The revision history then becomes an automatic source of checks that accumulates as the product changes.