Selecting Benchmarks for a Use Case
A benchmark score is only one piece of evidence. To turn evidence into a decision, you must decide which pieces to look at and how to weigh them together.
Why Published Scores Are a Starting Point
Suppose your team needs a language model for a product. You are rarely starting from zero. Many models have already been run against public benchmarks, and their scores are published. Those published results let you narrow a long list of candidates to a shortlist of models that look promising for your situation.
The problem is that there are too many benchmarks to examine them all. Benchmark results can help you identify promising models, but only if you decide which results to look at. This article follows the path from a pile of individual benchmark scores to an ordered ranking, which is the kind of ranking a leaderboard presents. Along the way it addresses the two decisions that shape that ranking: which benchmarks to include, and how to combine their results.
A benchmark score is only one piece of evidence. To turn evidence into a decision, you must decide which pieces to look at and how to weigh them together.
Starting From the Use Case
A useful way to think about relevance is to start from the use case rather than from the benchmark list. The benchmarks you choose should reflect what the model will be asked to do. A strong score on a task you never perform tells you little about your application.
The source gives two simple illustrations. If your application writes code, coding benchmarks are the natural center of your evaluation. If it must also avoid producing harmful text, a benchmark that measures toxicity becomes relevant too. Each requirement in the product points you toward a category of benchmark.
Choosing Benchmarks for a Code Assistant
A team is building a tool that suggests code to developers and must never produce abusive comments in its output. Which benchmarks belong in the evaluation?
- 1List what the product must do
The assistant writes code, so it needs a coding benchmark.
- 2List what the product must avoid
The assistant must not produce harmful text, so a toxicity benchmark is relevant.
- 3Filter out what the product never does
A benchmark about an unrelated skill, such as a task the assistant will never receive, adds evidence that does not bear on the decision and can be left out.
- 4Keep the matched set
The evaluation is built from the coding benchmark and the toxicity benchmark, because those reflect the tasks and behaviors the application needs.
The selected set is the coding benchmark plus the toxicity benchmark. The selection came from the use case, not from the length of the benchmark list.
Selection also shapes everything downstream. The benchmarks you include define the evidence that the final ranking is built from, so a leaderboard can only be as meaningful as the set of benchmarks behind it. A well-built ranking over the wrong benchmarks still answers the wrong question.
Tracing the Path From Scores to a Ranking
Read the following diagram as a pipeline. Candidate benchmarks are narrowed to the selected set. Each selected benchmark then produces results for every model. The aggregation method turns those per-benchmark results into the order that the leaderboard displays. Your judgment enters at two points: the filter step and the combine step.
Aggregation is the step that combines results across benchmarks to rank models. The source notes that there are multiple ways to do it, so the method is a real decision rather than a fixed procedure. A leaderboard is the ordered output of this aggregation. The source does not prescribe one specific aggregation formula, which is why the choice must be made deliberately and stated.
The Two Questions Behind Leaderboard Design
Leaderboard design rests on two questions. The first is which benchmarks to include. The second is how to aggregate their results. These questions are related, and the diagram below shows how. The benchmark set supplies the evidence, and the aggregation method decides how that evidence becomes an order.
The two decisions are not independent. The aggregation method can only operate on whatever results the inclusion decision produced, so the set of benchmarks comes first. A strong aggregation rule applied to an irrelevant set still yields an irrelevant ranking. Each subsection below examines one of these questions.
When a Model Wins One Benchmark and Loses Another
The difficulty is that a model can win on one benchmark and lose on another. The source describes this situation directly: a model does better on a coding benchmark but worse on a toxicity benchmark. A ranking has to place that model somewhere, so the evaluator must decide how much each result counts.
Consider two candidate models evaluated on the coding benchmark and the toxicity benchmark from the earlier example. Model A scores higher on the coding benchmark, while Model B scores higher on the toxicity benchmark. Whether Model A ranks above Model B depends entirely on the aggregation choice. If the method counts the two benchmarks equally, the outcome may be one order. If the team decides that avoiding harmful text matters more for its product, a different method can reverse the order. The scores did not change; the decision about how to combine them did.
Common mistake Treating the leaderboard order as a fact rather than a result of a method
Why it failsThe order depends on which benchmarks were included and how they were aggregated. Changing either can change the order.
Do this insteadReport the benchmark set and the aggregation method alongside the ranking.
Common mistake Including benchmarks because they are popular rather than because they match the use case
Why it failsA strong score on a task you never perform tells you little about your application.
Do this insteadStart from the tasks and behaviors the product requires, then pick the benchmarks that measure them.
Common mistake Assuming there is one correct way to combine benchmark results
Why it failsThe source notes that there are multiple ways to aggregate results, so the method is a real decision.
Do this insteadChoose a method deliberately and be able to explain why it reflects your priorities.
Borrowing From a Public Leaderboard
The source points to a practical shortcut. Public leaderboards already demonstrate how the two decisions can be made, so studying how they are built gives you a reference point. You do not have to design the process from nothing. You can borrow the logic of an existing public leaderboard, adapt its benchmark list to your use case, and adjust the aggregation to match your priorities.
| Part of a leaderboard | Public leaderboard | Your own leaderboard |
|---|---|---|
| Benchmark columns | The benchmarks the public board chose to include | The benchmarks matched to your tasks and behaviors |
| Aggregate score | The combined result produced by the board's aggregation method | The result of the aggregation method you chose and can justify |
| Ranking order | The ordered output the board displays | The ordered output of your own aggregation, used to shortlist models |
Mapping the parts of a public leaderboard onto a leaderboard built for your own use case
Practice: Narrowing a Candidate Shortlist
A team is choosing a language model for a customer support chat product. The product must answer questions accurately and must never produce abusive replies. Describe which kinds of benchmarks you would include and which you would filter out, and explain why. Then state one aggregation choice you would make and what it prioritizes.
Hints
- Start from the tasks and behaviors the product requires, not from the list of available benchmarks.
- A benchmark is relevant when it measures something the model will actually be asked to do or must avoid doing.
- Think about whether a model that excels at one required behavior but fails the other should rank high or low, and how your aggregation method would reflect that.
Key Takeaways
- There are too many benchmarks to examine them all, so a team must select a subset that matches its use case.
- Selection starts from what the product must do and must avoid, and the chosen benchmarks define the evidence behind the ranking.
- Aggregation combines results from several benchmarks into one leaderboard order; there are multiple methods, so the choice is part of the evaluation.
- The two questions of leaderboard design are which benchmarks to include and how to aggregate them, and the second depends on the first.
- Public leaderboards offer a reference design that you can adapt by adjusting the benchmark list and the aggregation to your priorities.