Designing Eval Sets for Real Distributions

Most engineers learn to trust software through one habit: write down what the system must do, write tests that check those requirements, and expect the tests to pass every time. For deterministic code this habit works very well. When a team begins building on a system whose behavior is judged statistically rather than absolutely, the same habit can start to work against them. The team keeps asking for a guarantee, and the approach they are using cannot supply one.

This article follows that friction. It looks at what a passing test actually means in each tradition, how an eval set relates to the real traffic a product receives, and what it looks like when a customer's own engineers start doing eval work themselves. Keep one question in mind as you read: when a test passes, what has been promised, and to whom?

Imagine a test case that checks one behavior of a system. In the spec-and-test tradition the test is written to hold for every run. If it passes, the team has confirmed a promise. If it fails, the system has broken that promise, and the failure is treated as a defect in the system.

In the eval discipline the same kind of case means something different. It is a measurement of a rate. The aim is for the case to pass with high probability across the distribution of inputs that matters to the product. A single pass or single fail says less than it appears to, because the question being answered is how often the system behaves acceptably across realistic traffic.

The diagram places the two traditions side by side. Both start from a test case, but they attach different promises to the same kind of result. Read down each column and notice where the promise lands: in the spec-and-test column it lands on every run, and in the eval column it lands on a rate across a distribution.

The source states the core distinction in one sentence: the spec-and-test tradition treats test cases as guarantees that always pass, while the eval discipline treats them as samples that should pass with high probability on the distribution that matters.

An eval set is not the full population of inputs a product will ever see. It is a collection of cases chosen to represent the distribution of inputs that matters. Each case sits inside that distribution as one sample, and the set as a whole is meant to be a usable picture of it.

Read the chain from the top. Production inputs are the whole stream of traffic. The distribution that matters is the part of that stream the product must handle well. Eval cases are samples taken from that distribution, and their pass rate is the measurement that stands in for behavior across it. The quality of the eval set therefore depends on how faithfully its cases reflect the distribution, not on how many cases there are.

Customer engineers trained in the spec-and-test tradition often request guarantees that an eval set cannot give. The source notes that the FDE must translate patiently, and that this translation is part of the customer's adoption process. The flowchart below sketches one reasoning path for such a request. The steps are a suggested pattern for the FDE, built from the source's framing rather than a fixed procedure stated in the source.

Early in an engagement the FDE carries most of the eval thinking. Over time the work is meant to move into the customer's hands. The source identifies one specific signal that marks this move. The customer has joined the work when their own engineers start producing eval material.

Two concrete behaviors count. The first is filing bugs that take the form of missing eval cases, meaning the engineer points out a region of the distribution the set does not cover. The second is submitting their own labeled examples. Either behavior shows the team is reasoning in eval terms rather than waiting for the outside engineer to supply every judgment.