Designing Evaluation Criteria and Scoring Rubrics

When two versions of an application are compared, the most natural move is to look for the one number that summarizes performance and pick the winner. One overall score feels clean, fair, and easy to defend. The belief behind this habit is that the overall number tells you how the system does for everyone, so a higher number means a better system across the board. This article tests that belief, because evaluation practice contains a case where it breaks down.

The overall column is only a summary. To see what it hides, read each group on its own: first the group-level percentages, then the raw counts that produced them. The source gives a concrete setup. Both models were tested on a set of 350 examples. Model A has 87 examples in Group 1 and 263 in Group 2. Model B has 270 examples in Group 1 and 80 in Group 2. The two models are therefore not measured on the same mix of cases, even though they share one test set.

This pattern is Simpson's paradox. A group-level advantage can disappear, or flip, when the groups are combined into one total. The source's explanation is that the overall figure is not an average that can be trusted to reflect each group, because the groups contain different numbers of examples. Model B has most of its examples in Group 1, where the scores are high. Model A has most of its examples in Group 2, where the scores are lower. The overall percentage mostly reflects where each model's examples happened to fall.

In this formula, n_g is the number of examples in group g and c_g is the number of those examples answered correctly. The overall score weights each group by its size, not by the importance of that group to your users. A model that is strong on a large group and weak on a small group can therefore look better overall than a model that is stronger on every group. Neither the per-group numbers nor the overall number is wrong on its own terms. The problem is treating the overall number as if it answered the question of which system is better for everyone.

The defense against Simpson's paradox is slicing. Slicing means separating your data into subsets and looking at your system's performance on each subset separately. Slicing is not a special technique reserved for paradoxes. It is a general habit that gives a finer-grained view of how the system behaves. Each slice is a subset of the evaluation set, and the per-slice results can be read individually and also combined into the single aggregate number.

Slices can be drawn from many dimensions. The source mentions paying users versus free users, traffic from mobile versus web, and usage patterns. Each of these splits can reveal a group where performance differs from the total. Slicing only helps when you choose slices that matter for your users, so the choice of dimension is part of the evaluation design rather than an afterthought.

A single evaluation set rarely answers every question, so the source recommends maintaining several sets, each with a different purpose. One set should represent the distribution of actual production data, so you can estimate how the system does overall. Other sets should represent specific slices, so you can see how it performs on the cases that matter most. The source states the principle directly: if you care about something, put a test set on it.

Different criteria often need different evaluation methods. The source gives three examples. A small, specialized toxicity classifier suits toxicity detection. Semantic similarity can measure relevance between the response and the user's original question. An AI judge can measure factual consistency between the response and the whole context. Specialized scorers and AI judges both depend on an unambiguous scoring rubric and on examples to work well.

Methods can also be mixed for the same criterion. A cheap classifier can give a low-quality signal on all of your data, while an expensive AI judge gives a high-quality signal on a small fraction, such as one percent. This combination gives a reasonable level of confidence while keeping costs manageable. When logprobs are available, the source recommends using them, because they measure how confident a model is about a generated token. This is especially useful for classification: if the three class probabilities are each between 30 and 40 percent, the model is not confident, and if one class is at 95 percent, the model is highly confident.

An evaluation method needs data with trustworthy labels to score against. You need annotated data for each component of your system and for each criterion, whether the application is evaluated turn by turn in a conversation or as a whole task. The build process follows a sequence of steps, and the scoring rubric matters at every one of them.

The rubric is the link between the criterion and the label. A rubric defines what each score level means for a criterion, and it is the document labelers and automated judges both rely on. The source does not describe a further adjudication stage in this material, so the process above is the sequence to keep in mind: sample, label against a clear rubric, and keep the guideline for reuse.

An evaluation set should be large enough that its result is reliable, but small enough to remain affordable to run. The source offers a practical check using bootstraps. Imagine an evaluation set of 100 examples, and you want to know whether a different set of 100 examples would produce a different result. You build bootstrap samples by drawing from your set with replacement, then score each one.

Suppose one bootstrap gives 90 percent and another gives 70 percent. Those two results come from resampling the same 100-example set, and they differ by 20 points. The source's reading is that the evaluation pipeline is not trustworthy at that size. Bootstrapping tests reliability, not the truth of any single number. Stable bootstrap results suggest the set is large enough for this purpose, while wildly varying results are a signal to collect more examples.

Sample size also matters when comparing two systems. Suppose a new prompt scores 10 percent higher than the old one. In theory, a statistical significance test can compute the sample size needed for a given confidence level, such as 95 percent, if you know the score distribution. In practice that distribution is hard to know, so the source offers a rough guide from OpenAI. Its rule of thumb is that for every threefold decrease in the score difference, the number of samples needed increases about tenfold. Small differences therefore demand much larger evaluation sets.

For reference, the source notes that among benchmarks in Eleuther's lm-evaluation-harness, the median benchmark has 1,000 examples and the average has 2,159. The organizers of the Inverse Scaling prize suggested 300 examples as an absolute minimum and preferred at least 1,000, especially when examples are synthesized.

Trace the evaluation chain in order. Start with the overall score, then read every subgroup to see whether the ranking holds. Slice the evaluation set along dimensions that matter to users, and keep several sets for different purposes. Choose a method that fits each criterion, mix cheap and expensive signals where that helps, and build annotated data against a clear rubric. Before trusting the result, run bootstraps to check stability and consider how many examples a small difference would require.