Benchmark Selection

Suppose you are building an application and you want a foundation model that does it well. A natural first step is to open a public leaderboard, find the model at the top, and start building. The shortcut is tempting because a leaderboard presents a single ranking, and a single ranking looks authoritative. It rests on a belief that deserves a careful look: that a leaderboard rank summarizes how well a model will perform on your task.

This article examines why that belief breaks down. Along the way it covers how leaderboards choose their benchmarks, how to read the relationships between benchmarks, how scores get combined into one number, and how to build a benchmark set that reflects your own needs. Before reading on, make a prediction about the situation below.

Public leaderboards rank models by their aggregated performance on a subset of benchmarks. They are genuinely useful, but they are far from comprehensive, and the first reason is compute. Evaluating a model on a benchmark requires compute, so most leaderboards can include only a small number of benchmarks. That constraint forces a choice, and a leaderboard may leave out an important benchmark simply because it is expensive to run.

Two examples show this constraint at work. HELM Lite left out an information retrieval benchmark called MS MARCO because it is expensive to run. Hugging Face chose not to include HumanEval, a coding benchmark, because it has large compute requirements, since it needs a great many generated completions. In both cases the omission is a practical decision, not a judgment that the skill does not matter.

Leaderboard size has also grown only slowly. When Hugging Face first launched its Open LLM Leaderboard in 2023, it consisted of four benchmarks. By the end of that year it had been extended to six. Even the larger set is small compared with the range of things foundation models are asked to do. A small set of benchmarks is not nearly enough to represent the broad capabilities and the different failure modes of these models.

Leaderboard designers generally try to balance coverage against the number of benchmarks. They aim for a small set that spans a wide range of capabilities, such as reasoning, factual consistency, and domain-specific skills like math and science. At a high level this approach makes sense. The difficulty is that coverage is rarely defined precisely, and the public descriptions do not explain why a leaderboard stops at six benchmarks or at ten.

Different leaderboards do not share the same benchmarks. Stanford's HELM Leaderboard used ten benchmarks around the same time that Hugging Face's leaderboard used six. Only two benchmarks appeared on both: MMLU and GSM-8K. That overlap is small, and it makes the rankings hard to compare and interpret. A model can rank highly on one leaderboard and lower on the other without either ranking being wrong, because each one measures a different set of tasks.

Read the diagram as two columns that share only a small middle. Each leaderboard ranks models using its own column of benchmarks. When a model's strengths sit in the benchmarks that only one leaderboard uses, its position on that leaderboard can look very different from its position on the other. The difference comes from which tests were chosen, not from any single model result.

An aspect of benchmark selection that is often overlooked is benchmark correlation. Benchmark correlation describes how similar the performance of two benchmarks is across models. If two benchmarks tend to rise and fall together, they measure similar things. This matters for selection for a simple reason: if two benchmarks are perfectly correlated, including both adds nothing, so you do not want both of them. Strongly correlated benchmarks can also exaggerate biases, because the same tendency is counted twice.

The correlation values discussed here are Pearson correlations among the six benchmarks on the Hugging Face leaderboard, computed in January 2024. A value near 1.0 means two benchmarks are strongly related. Each benchmark has a correlation of exactly 1.0 with itself, which is why the diagonal of such a table is always 1.0. The values off the diagonal are the ones to read.

The reasoning cluster is consistent with the fact that all three benchmarks test reasoning capabilities. Including ARC-C, MMLU, and WinoGrande together largely repeats one signal. TruthfulQA sits apart, with correlations around 0.45 to 0.55. The source's reading of this pattern is that improving a model's reasoning and math capabilities does not always improve its truthfulness. Benchmarks that look different on the surface can therefore carry very different information, and benchmarks that look similar may be redundant.

Once benchmarks are chosen, their results must be combined into a ranking. This combining step is called aggregation. Hugging Face averaged a model's scores on all its benchmarks to produce the final score used for ranking. Follow one model through that process. Each benchmark result enters the average at equal strength, and the average becomes the single number that places the model on the leaderboard.

Averaging has a hidden assumption. It treats all benchmark scores equally, so a given percentage counts the same no matter which benchmark produced it. An 80% score on TruthfulQA is treated the same as an 80% score on GSM-8K, even though an 80% on TruthfulQA may be much harder to achieve. Averaging also gives every benchmark the same weight, even when some tasks matter much more to a particular purpose. If truthfulness matters far more than grade school math for your application, an equal-weight average does not reflect that.

The HELM authors chose a different approach. Rather than averaging, they used mean win rate, which they defined as the fraction of times a model obtains a better score than another model, averaged across scenarios. This changes the question from how high a model scored to how often it beat its competitors. Both methods compress many results into one number, and each one discards different information.

The correct approach begins by treating your selection as building a private leaderboard. Instead of ranking models by a generic criterion, you rank them by criteria you choose. The first step is to gather a list of benchmarks that evaluate the capabilities important to your application. The benchmark list is the foundation of the leaderboard, so it should follow the work your application does rather than the work a public leaderboard happened to include.

The reliability check deserves particular attention. A benchmark with an impressive name is not automatically a good test for your needs. Ask what the benchmark actually measures and whether that matches the behavior you care about. Matching is the step that turns a list of popular tests into a set of benchmarks that speak to your product.

Evaluation is harder than it looks, and the difficulty shows up in a recurring observation: each time OpenAI updates its models, users complain that they seem to be getting worse. A study by Stanford and UC Berkeley (Chen et al., 2023) found that for many benchmarks, the performance of both GPT-3.5 and GPT-4 changed significantly between March 2023 and June 2023. This is score drift: the same model on the same benchmark can produce different numbers at different times.

The source offers a careful reading of why this perception exists. One potential explanation is that evaluation is hard, and nobody, not even the model provider, knows for certain whether a model is improving or declining. The source adds that while evaluation is clearly difficult, it doubts that OpenAI would work completely blind. Whatever the cause, the practical lesson holds: a score you measured at one moment may not describe the model you use later.

Missing scores are a second challenge. Not all models have publicly available scores on all benchmarks. If the model you care about has no published score on your benchmark, you will need to run the evaluation yourself. An evaluation harness can help with that work, meaning a tool that organizes running a model against a benchmark and collecting its results.

Evaluation cost is a third challenge. The source gives one concrete figure: Stanford spent approximately 80,000 to 100,000 dollars to evaluate 30 models on its full HELM suite. The more models you want to compare and the more benchmarks you want to use, the more expensive the process becomes. Cost is therefore a practical constraint that shapes how many models and benchmarks you can afford to test in full.

Mixed scoring units are the fourth challenge. One benchmark might report accuracy, another F1, and another BLEU score. These numbers do not share a common unit or scale, so you cannot simply add them together and expect the result to mean something. Scores must be handled before they can be combined, which is the job of the weighting step described next.

After you have chosen a set of benchmarks and collected scores for the models you care about, you must combine those scores into a ranking. The source's guidance is to decide how important each benchmark is to your application and weigh the scores accordingly. Weighting makes your priorities explicit: a benchmark central to your product counts for more than one that is only loosely related. The diagram below shows the flow from individual benchmark scores to a weighted ranking.

The following example is an invented illustration of the weighting idea, not a figure from the source. Both scores are already expressed on the same zero-to-one scale. That assumption matters, because real benchmark scores often are not on a shared scale and must be converted before they can be compared. The weighted score for a model is the sum of each benchmark score multiplied by its weight, where the weights add up to 1.