Evaluation Pipelines and Reliability

Once you decide to measure something, you need a set of examples to measure it on. That set is called an evaluation set, sometimes called a test set. The examples in it are curated and annotated for evaluation. They also serve a second purpose: later, the same data can be used to synthesize more data for training. The practical question this article addresses is how many examples an evaluation set should contain.

There is no single correct number. The right size depends on the application and on the evaluation methods you use. Sizing the set is a decision about where to land between two pressures. The set must be large enough that the result is reliable, meaning it reflects the model's real behavior rather than a lucky or unlucky draw of examples. The set must also be small enough to be affordable, because every evaluation pass costs time and money.

Suppose you have an evaluation set of 100 examples. A natural question is whether 100 is enough for the result to be reliable. You cannot simply collect a second independent set of 100 examples every time you want to check, so the method has to work with the data you already have.

The underlying question is a thought experiment. If you evaluated the model on a different evaluation set of 100 examples, would you get a different result? Bootstrapping is a practical stand-in for that different set. Keep this question in mind as you read the next section, because the bootstrap is designed to answer it using only the 100 examples you already have.

Bootstrapping resamples the evaluation set with replacement, evaluates the model on each resample, and uses the spread of the results to judge whether the set is large enough. Each bootstrap is a new set of examples drawn from your original set. Because the drawing is done with replacement, the same example can appear more than once in a single bootstrap, and some examples from the original set may be left out of it entirely.

The underlying logic is that the bootstrap results are not the answer by themselves. Their spread is the signal. A narrow spread means the evaluation set gives the model a consistent verdict. A wide spread means the verdict depends too heavily on which examples happened to be included, which is the situation that calls for more evaluation data. The diagram above shows two resamples only to make the idea concrete. In practice you run many resamples and look at the whole spread.

Stability is only half of the problem. Evaluation results are also used to compare systems, such as a new prompt against an old one. Imagine a new prompt scores 10% higher than the old prompt. How large must the evaluation set be before you can be certain the new prompt is really better, rather than merely looking better on this particular set?

In principle, a statistical significance test answers this. Given a level of confidence, such as 95% confidence, the test can compute the sample size needed, provided you know the score distribution. In practice the true score distribution is hard to know, so the exact computation is rarely available. That gap is why practitioners fall back on rough estimates.

OpenAI has offered a rough estimate of how many evaluation samples you need to be reasonably certain that one system is better, given the score difference you want to detect. The rule of thumb that comes with it is this: for every 3 times decrease in the score difference, the number of samples needed increases about 10 times.

The rough estimates for 95% confidence follow the rule of thumb. Each row gives the score difference you want to detect and the approximate number of samples needed to detect it. Read the table by first locating the difference you actually care about, then reading across to the sample count. Notice that the rows get more expensive as the difference shrinks.

The table makes one pattern clear. Detecting big differences is cheap, and detecting small differences is expensive. A 30% difference can be confirmed with roughly 10 samples, while a 1% difference can require roughly 10,000. Before you decide how large an evaluation set to build, decide how small a difference you actually need to be able to detect. A 10% difference needs about 100 samples, and a 3% difference needs about 1,000.

It helps to know what other evaluation sets look like. Among evaluation benchmarks in Eleuther's lm-evaluation-harness, the median benchmark has 1,000 examples, and the average is 2,159. These figures give a sense of what established benchmarks typically use.

The organizers of the Inverse Scaling prize offered a different kind of guidance. They suggested that 300 examples is the absolute minimum, and they would prefer at least 1,000, especially when the examples are synthesized rather than collected from natural sources. Benchmarks are a reference point, not a target. The right size for your own set still depends on the stability and comparison questions described above.