AI-Generated Data for Evaluation
Annotated examples are needed to evaluate each component and criterion, and production data is preferred when available. Humans or AI can supply labels when natural labels are absent.
Why One Overall Score Can Mislead
Suppose you are choosing between two models for an application. You test both on your annotated examples, and one model comes out ahead on the overall score. It seems natural to assume that the same model is stronger everywhere, and that a single number is enough to decide. This article tests that assumption. Annotating evaluation data well means more than producing one score; it means seeing how the system performs on the different kinds of data it will actually meet.
What do you think happens?
Model A is better than Model B on every part of the data. Must Model A also be better on the data taken as a whole?
Reveal answer
No. The overall result can point the other way. This is Simpson's paradox.
The belief feels airtight because each part points the same way, but the overall total weights the parts differently for each model. The numbers in the next section show exactly how this happens.
Tracing Two Models Across Two Groups
Consider two models evaluated on the same 350 annotated examples, split into two groups. Model A gets 81 of 87 correct in Group 1, 192 of 263 correct in Group 2, and 273 of 350 correct overall. Model B gets 234 of 270 correct in Group 1, 55 of 80 correct in Group 2, and 289 of 350 correct overall. The table lays these figures out side by side.
| Measure | Model A | Model B |
|---|---|---|
| Group 1 accuracy | 93% (81 of 87) | 87% (234 of 270) |
| Group 2 accuracy | 73% (192 of 263) | 69% (55 of 80) |
| Overall accuracy | 78% (273 of 350) | 83% (289 of 350) |
Two models on the same 350 annotated examples, split into two groups
Finding the Reversal
Which model is stronger within each group, and which model wins overall?
- 1Compare Group 1
Model A reaches 93% against Model B's 87%. Model A is ahead, with $\frac{81}{87} \approx 93\%$ versus $\frac{234}{270} \approx 87\%$.
- 2Compare Group 2
Model A reaches 73% against Model B's 69%. Model A is ahead again, with $\frac{192}{263} \approx 73\%$ versus $\frac{55}{80} \approx 69\%$.
- 3Compare the overall score
Model A reaches 78% against Model B's 83%. Model B is ahead, with $\frac{273}{350} \approx 78\%$ versus $\frac{289}{350} \approx 83\%$.
- 4Check how the examples are distributed
Model A has 263 of its 350 examples in Group 2, which is about 75%. Model B has 270 of its 350 in Group 1, which is about 77%. Both models score lower on Group 2, so the overall total mostly reflects different groups for each model.
Model A wins in every group but loses the overall comparison. The single overall number reverses the result shown in each subgroup.
Why the Weights Cause the Reversal
Simpson's paradox is the pattern in which one model beats another on every subgroup but loses on the aggregated data. The cause is that the groups are weighted differently between the models being compared.
The reversal does not come from the per-group results, which all favor Model A. It comes from the weights. The overall score is a combined total, and each model contributes a different mix of examples from each group. When one model has most of its examples in a group where scores are low for both models, its combined total is pulled down, even though it is stronger within that group.
The source notes that the same phenomenon runs in the other direction too. A trend can appear within several different groups and then disappear, or reverse, once the groups are combined. In both cases the combined number is not a reliable stand-in for what happens within the parts.
Annotated Examples for Every Component
All of this depends on having annotated data in the first place. The source says you need annotated data to evaluate each of your system's components and each criterion, for both turn-based and task-based evaluation. An application with several parts needs labeled examples that let you judge each part, not only the final output.
- Use actual production data where possible, since production data is preferred when it is available.
- If your application has natural labels, use them.
- If natural labels are absent, humans or AI can supply the labels.
- Make the criteria used to label each example clearly defined, because the success of annotation depends on the clarity of the scoring rubric.
- Reuse the annotation guideline you write for evaluation later as instruction data for finetuning, if you decide to finetune your model.
Data Slicing and Its Purposes
The remedy for a misleading overall score is data slicing. Slicing means separating your data into subsets and looking at your system's performance on each subset separately. The key word is separately. A slice is not a different evaluation; it is the same evaluation restricted to a portion of the data, so that patterns hidden in the combined total become visible.
The source gives several reasons a finer-grained view is worth the effort. Each one is a purpose a slice can serve.
- Avoiding bias, such as bias against minority user groups.
- Debugging: if the application performs poorly on a subset, the cause may be an attribute of that subset, such as its length, topic, or format.
- Finding areas for improvement: if the application is weak on long inputs, you might try a different processing technique or a newer model that handles long inputs better.
- Guarding against Simpson's paradox, because a slice shows the within-group result that a combined total can hide.
Mistakes When Reading Slices
Common mistake Choosing a model by its overall score alone
What it looks likeModel A at 78% overall loses to Model B at 83%, so Model B is picked without looking at the groups.
Why it failsModel A is ahead in both groups, so the total reverses what every part shows.
Do this insteadReport the score for each slice next to the overall score and check whether the two agree.
Common mistake Comparing overall totals when the models have different group mixes
What it looks likeModel A has about 75% of its examples in Group 2, while Model B has about 77% of its examples in Group 1.
Why it failsThe totals are weighted by different groups, so they do not compare the models on equal terms.
Do this insteadCount how many examples each model has in each slice before interpreting the overall figure.
Common mistake Treating a slice as its own evaluation with its own rules
Why it failsA slice is the same evaluation restricted to part of the data, so the labels and criteria must stay the same.
Do this insteadKeep the same annotation guideline and scoring rubric across every slice.
Designing Several Evaluation Sets
Slicing becomes practical when you organize your annotated data into several evaluation sets, each serving a distinct purpose. The source recommends building several sets that cover the production distribution, tiers or traffic slices, frequent mistakes, typical user errors, and out-of-scope inputs. Each set answers a different question, so no single set has to do everything.
Read the diagram as a way of deciding what each set is for. The production traffic set asks how the system performs on the kinds of data it will realistically meet. The known failure set asks whether the system still struggles on frequent mistakes and typical user errors. The out-of-scope set asks how the system behaves on inputs at the edge of its intended use. Together they keep production realism, known weak spots, and out-of-scope inputs in view at the same time.
Planning a Slice-Aware Evaluation
You are comparing two models for a customer-support application. You have 400 annotated examples. Plan three evaluation sets: one for production traffic, one for known failure cases, and one for out-of-scope inputs. Then list which slices you would report separately, and what you would check before trusting an overall winner.
Hints
- Start from the purposes of each set: realistic performance, weak spots, and boundaries.
- Ask whether each model's examples are spread evenly across the slices before comparing totals.
- Make sure every set uses the same annotation criteria so the results can be compared.
Key Takeaways
- Annotated examples are needed to evaluate each component and each criterion of an application, and production data is preferred when it is available.
- Simpson's paradox occurs when one model beats another on every subgroup but loses on the aggregated data, because the groups are weighted differently between the models.
- Data slicing separates data into subsets and evaluates performance on each one separately, which helps with bias, debugging, improvement, and avoiding Simpson's paradox.
- Organizing annotated examples into several evaluation sets covers production traffic, known failure cases, and out-of-scope inputs, with each set answering a different question.