Building Your Own Evaluation Benchmarks

Public leaderboards make model selection look easy. You open a ranking, see which model sits at the top, and assume that model should power your product. This shortcut is the source of many selection errors, and it is worth examining carefully before you build anything.

The rest of this article treats model selection as a small exercise. You will see the belief that causes most selection errors, learn how to build a shortlist of benchmarks, handle the practical problems that come with scores, and combine several scores into one ranking.

The misconception to watch for is this: if a model is ranked first overall, it is the right model for my application. This feels reasonable because a general ranking seems to summarize quality. The correct approach is to stop reading a leaderboard as a verdict and start reading it as one input. The leaderboard answers one question. Your selection needs a different question answered.

The correct approach begins by treating your selection as building a private leaderboard. Instead of ranking models by a generic criterion, you rank them by criteria you choose. Each step of the process then serves that private ranking: first you choose benchmarks, then you collect scores, then you combine those scores with weights that reflect what matters to your product.

The first concrete step is to gather a list of benchmarks that evaluate the capabilities important to your application. The benchmark list is the foundation of your private leaderboard, so it should follow the work your application does. If your product summarizes documents, benchmarks that never touch summarization or reading comprehension tell you little about it. If your product generates text that is compared against reference text, metrics such as BLEU are relevant, while a benchmark on a different skill is not.

Evaluation is harder than it looks. A recurring observation is that each time OpenAI updates its models, users complain that they seem to be getting worse. A study by Stanford and UC Berkeley (Chen et al., 2023) examined this. It found that for many benchmarks, the performance of both GPT-3.5 and GPT-4 changed significantly between March 2023 and June 2023.

The source offers a careful reading of why this perception exists. One explanation is that evaluation is hard, and nobody, not even the model provider, knows for certain whether a model is improving or declining. The source adds that while evaluation is clearly difficult, it doubts that OpenAI would work completely blind. Whatever the cause, the practical lesson holds: a score you measured at one moment may not describe the model you use later. Benchmarks, their data, and the evaluation setup can all change, so a score is a snapshot.

Public scores do not cover everything. Not all models have publicly available scores on all benchmarks. If the model you care about has no published score on a benchmark you need, you will have to run that evaluation yourself. An evaluation harness can help: it is a tool that organizes running a model against a benchmark and collecting its results.

Running evaluations also costs money, and the cost grows quickly. The source gives one concrete figure: Stanford spent approximately 80,000 to 100,000 dollars to evaluate 30 models on its full HELM suite. The more models you compare and the more benchmarks you use, the more expensive the process becomes. Cost is therefore a practical constraint that limits how many models and benchmarks you can afford to test in full.

After you have chosen benchmarks and collected scores, you must combine them into a ranking. This step is not automatic, because the scores do not share a common unit or scale. One benchmark might report accuracy, another F1, and another BLEU score. You cannot simply add these numbers together and expect the result to mean something. The source warns that real benchmark scores often are not on a shared scale and must be handled before they can be compared.

The source's guidance is to decide how important each benchmark is to your application and weigh the scores accordingly. Weighting makes your priorities explicit. A benchmark central to your product counts for more than one that is only loosely related. The diagram below shows the flow from individual benchmark scores to a single weighted ranking.

The following example is an invented illustration of the weighting idea, not a figure from the source. Suppose your application cares more about accuracy than about F1, so you give accuracy a weight of 0.6 and F1 a weight of 0.4. Both scores are already expressed on the same zero-to-one scale. That assumption matters, because the source warns that real scores often are not on a shared scale.

The example shows that the weights decide the outcome. Model A has the higher accuracy, yet Model B wins, because the weighted total depends on both scores in the proportions you chose. Changing the weights could reverse the order, which is why the weights should reflect a deliberate judgment about your application.