Designing Evaluation Sets

Once you decide to measure something, you need a set of examples to measure it on. That set is called an evaluation set, sometimes called a test set. Its examples are curated and annotated for evaluation. They also serve a second purpose: later, the same data can be used to synthesize more data for training. This article answers one question: how many examples should an evaluation set contain?

There is no single correct number. The right size depends on the application and on the evaluation methods you use. Before reading on, pause and ask yourself: if you scored a model on a different set of 100 examples, would you get the same score? Hold on to that question, because the rest of the article is built around it.

The guiding principle is a balance between two pressures. The first is reliability. The set must be large enough that the evaluation result reflects the model's real behavior, and not a lucky or unlucky draw of examples. The second is affordability. The set must be small enough that it is not prohibitively expensive to run, because every evaluation pass costs time and money. Sizing the set is a decision about where to land between these two limits.

Suppose you have an evaluation set of 100 examples. A natural question is whether 100 is enough for the result to be reliable. You cannot simply collect a second independent set of 100 examples each time you want to check, so the method works with the data you already have. You create multiple bootstraps of those 100 examples and see whether they give similar evaluation results.

A bootstrap is a resample of the evaluation set drawn with replacement. Each resample has the same number of examples as the original, but some examples appear more than once and others are left out entirely. Because of this, each bootstrap is a practical stand-in for a different evaluation set. The underlying thought experiment is the same as before: would a different set of examples have produced a different result?

The bootstrap results are not the answer by themselves. Their spread is the signal. A narrow spread means the evaluation set is giving the model a consistent verdict. A wide spread means the verdict depends too much on which examples were included, which is the situation that calls for more evaluation data.

The bootstrap check ends in a yes or no decision. If the spread is narrow, the set is stable enough to trust for this purpose. If it is wide, the set is too small for a reliable answer, and you should add examples and repeat the check.

Stability is only half of the problem. Evaluation results are also used to compare systems, to decide which model, prompt, or other component is better. Imagine a new prompt scores 10% higher than the old prompt. How large must the evaluation set be before you can be certain the new prompt is really better, rather than just looking better on this particular set?

In principle, a statistical significance test answers this. Given a level of confidence, such as 95%, the test can compute the sample size needed, provided you know the score distribution. In practice, the true score distribution is hard to know, so the exact computation is rarely available. That gap is why practitioners fall back on rough estimates.

OpenAI has offered a rough estimate of how many evaluation samples you need to be reasonably certain that one system is better, given the score difference you want to detect. A useful rule of thumb comes with it: for every 3 times decrease in the score difference, the number of samples needed increases about 10 times. The table below captures these estimates for 95% confidence.

The table makes one pattern clear. Detecting big differences is cheap, and detecting small differences is expensive. A 30% difference can be confirmed with roughly 10 samples, while a 1% difference can require roughly 10,000. Before you decide how large an evaluation set to build, decide how small a difference you actually need to be able to detect.

It helps to know what other evaluation sets look like. Among the evaluation benchmarks in Eleuther's lm-evaluation-harness, the median benchmark has 1,000 examples, and the average is 2,159. Those figures give a sense of what established benchmarks typically use.

The organizers of the Inverse Scaling prize offered different guidance. They suggested that 300 examples is the absolute minimum, and they would prefer at least 1,000, especially when the examples are synthesized rather than collected from natural sources. Benchmarks are a reference point, not a target. A set of 300 is the floor the organizers named, and a set below 1,000 falls under the preferred size they described. The right size for your own set still depends on the stability and comparison questions covered above.