Aggregating Model Evaluation Results

Suppose you need a language model for a product. You are rarely starting from zero. Many models have already been run against public benchmarks, and their scores are published. Those published results let you narrow a long list of candidates to a shortlist of models that look promising for your situation. The process moves from a pile of individual benchmark scores to an ordered ranking, which is the kind of ranking a leaderboard presents. This article follows that path, step by step, and shows where your own judgment has to enter.

Benchmark results can help you identify promising models, but there are too many benchmarks to examine them all. Selection is therefore required. A team that tried to read every published result would spend its effort on tests that have nothing to do with the work it actually needs done, and it would still have no clear answer at the end.

Selection is not a convenience. It determines the evidence behind every later step. The benchmarks you include define what the final ranking is built from, so a leaderboard can only be as meaningful as the set of benchmarks behind it. A carefully ordered ranking built on the wrong tests is still a poor guide to a decision.

A useful way to think about relevance is to start from the use case rather than from the benchmark list. Ask what the model will be asked to do. If your application writes code, coding benchmarks are the natural center of your evaluation. If it must also avoid producing harmful text, a benchmark that measures toxicity becomes relevant too. The benchmarks should reflect the tasks and behaviors your application needs, because a strong score on a task you never perform tells you little about your application.

Read the process as a pipeline. Candidate benchmarks are narrowed to a selected set. Each selected benchmark then produces results for every model. The aggregation method turns those per-benchmark results into the order that the leaderboard displays. Follow a single model through the stages below and notice where your judgment enters.

In the diagram, the first judgment happens at the filter step, where you decide which benchmarks count as evidence. The second happens at the combine step, where you decide how the per-benchmark results are merged. Changing either decision can change the final order even when the underlying scores stay exactly the same.

The difficulty of aggregation is that a model can win on one benchmark and lose on another. The source describes exactly this situation: a model that does better on a coding benchmark but worse on a toxicity benchmark. Aggregation forces you to resolve that tension, because a ranking must place each model somewhere. There is no single fixed rule that settles every such case, and the choice of method is part of the evaluation, not a neutral step.

Leaderboard design is driven by two questions. The first is which benchmarks to include. The second is how to aggregate their results. The two are linked: the benchmarks chosen define the evidence, and the aggregation method defines how that evidence becomes an order. Both feed into the final leaderboard, and neither can be skipped.

Because aggregation is a real decision with several possible methods, the method cannot be treated as a formality. A ranking reflects both the evidence and the rule used to merge it. When you read or build a leaderboard, ask both questions explicitly.

You do not have to design the process from nothing. Public leaderboards already demonstrate how these two decisions can be made, so studying how they are built gives you a reference point. Borrowing their logic is a practical shortcut, but the borrowing has to be deliberate: some parts carry over directly, and others must be changed to fit your situation.