Evaluating AI Products with Eval Suites

Imagine two teams that ship the same AI assistant. Both watch a dashboard full of averages. Both see the numbers drift slightly downward in the third week after launch. The first team responds by tweaking settings and hoping the averages recover. The second team opens a handful of individual conversations, reads what the user actually typed and what the assistant actually answered, and discovers that one kind of request is being handled badly again and again. The second team has begun working through the per-trace loop.

This article follows that second team's work. It explains how the loop is organized, how a pattern is pulled out of several exchanges, how that pattern becomes a test the team can run again, and why this cycle is what turns a fragile launch into a product people keep using.

The discipline asks for something that feels slow. You read the input the user supplied and the output the product produced, in their exact wording. Summary numbers compress many exchanges into one figure, and that compression hides the specific words where a failure lives. A dashboard can tell you that quality dipped; it cannot tell you which request caused the dip or what the product said in response. Reading the raw exchange keeps those words visible, which is why the loop begins there.

It helps to see the loop as a sequence in which each stage produces something the next stage uses. Reading sampled traces produces a hypothesis. The hypothesis is encoded as an eval case. The eval case is added to the standing eval suite. The suite is then used to grade a proposed fix. The fix is not the end of the chain: its grade comes from the suite, which already holds everything the team has learned so far. After grading, the team reads traces again, and the cycle repeats.

The return edge is what makes the loop cumulative. Each pass leaves behind a new test, so the next pass of reading starts from a larger body of knowledge and a larger suite than the last one did.

A single trace rarely tells you enough by itself. The loop asks you to read enough traces in sequence to form a working hypothesis about what is going wrong. The hypothesis is therefore a pattern you notice across several exchanges, not a conclusion drawn from one unusual reply. One odd answer might be a fluke; the same kind of request mishandled across several exchanges points to a shared cause.

A hypothesis is useful only if it can be checked again later. The loop therefore encodes it as an eval case: a test built from the hypothesis that captures the kind of input that exposed the problem and the behavior a good answer should show. Writing the case is what converts an observation made while reading into something the team can run repeatedly.

The eval case then becomes part of the standing eval suite. The word standing matters. The suite is not rebuilt for each change; it accumulates. A new case written from the hypothesis above would include a feature-specific request like the ones that exposed the problem, along with a description of what a good answer must do, which is address the named feature rather than the general category.

When a fix is proposed, it is graded against the suite. The fix must handle the new case and must also keep passing the cases the team wrote earlier. The suite therefore acts as a gate: a proposed change is checked against everything already learned, not only against the failure that prompted it. A fix that solves the new case but breaks an older one has not passed the suite.

Repeating this loop is what produces week-over-week improvement. Each cycle of reading traces, forming a hypothesis, encoding it, and grading a fix adds to what the team knows and what the suite checks. Later fixes are graded against everything already learned, so gains made in earlier weeks are protected rather than quietly lost. The source describes these improvement curves as the thing that separates a deployment customers keep using from one they quietly turn off about six weeks after launch.