Public Benchmarks and Their Limits

A top rank on a general public leaderboard does not guarantee the best model for your application, so start by matching benchmarks to the capabilities you need.

Programming

Why the Top Rank Can Mislead

Public leaderboards make model selection look easy. You open a ranking, see which model sits at the top, and assume that model should power your product. This shortcut is the most common source of selection errors, and this article examines it closely.

A leaderboard answers one question: which model performs best on a particular set of public tests. Your question is different: which model performs best on the tasks your application actually performs, measured in a way you trust. Those two questions often have different answers.

The article follows a simple path. You will first meet the belief that causes most selection errors, then see how to build a shortlist of benchmarks, then learn the practical problems that come with scores, and finally combine several scores into one ranking.

Leaderboard Rank Versus Application Fit

The misconception to watch for is this: if a model is ranked first overall, it is the right model for my application. This feels reasonable, because a general ranking seems to summarize quality. The source material pushes back on this directly. The best model overall might not be the best model for your application.

The reason is that a general ranking averages across many kinds of tasks, while your application may depend heavily on one or two of them. A model that is strong on the average can still be weak on the exact task your product performs, and a model that is slightly lower overall can be the stronger choice for that task.

Building a Private Leaderboard

The source's recommended approach is to treat your selection as building a private leaderboard. Instead of ranking models by a generic criterion, you rank them by criteria you choose. The benchmark list is the foundation of that leaderboard, so it should follow the work your application does.

1
List the capabilities your application needs, described in terms of the work it performs.
2
Find benchmarks that evaluate those capabilities, and set aside any that do not match your needs.
3
Check whether each chosen benchmark is reliable.
4
Collect scores for the models you care about, noting any that are missing.
5
Combine the scores into one ranking using weights that reflect your priorities.

Matching Benchmarks to Capabilities

Matching starts from the capabilities your application needs. Each needed capability should point to one or more benchmarks that measure it. The concept map below shows a small example: a product that answers questions, writes code, and summarizes contracts. Two of those capabilities have a benchmark behind them, and one does not.

Score Drift, Missing Scores, and Cost

Scores drift over time. The source points to a study by Stanford and UC Berkeley (Chen et al., 2023), which found that for many benchmarks, the performance of both GPT-3.5 and GPT-4 changed significantly between March 2023 and June 2023. This matches a recurring observation that users feel models get worse each time the provider updates them. The source offers a careful reading: evaluation is hard, and nobody, not even the model provider, knows for certain whether a model is improving or declining. It also doubts that OpenAI would work completely blind.

The practical lesson holds whatever the cause: a score you measured at one moment may not describe the model you use later. A rank you recorded last year may no longer hold, so dated scores should be treated as provisional.

Scores can also be missing. Not all models have publicly available scores on all benchmarks. If the model you care about has no published score on your benchmark, you will need to run the evaluation yourself. An evaluation harness, meaning a tool that organizes running a model against a benchmark and collecting its results, can help with that work.

Running evaluations costs money, and the cost grows quickly. The source gives one concrete figure: Stanford spent approximately 80,000 to 100,000 dollars to evaluate 30 models on its full HELM suite. The more models and benchmarks you want to compare, the more expensive the process becomes, so cost limits how many models you can test in full.

Putting Mixed Units on One Scale

After you have chosen benchmarks and collected scores, you must combine them into a ranking. This step is not automatic, because the scores do not share a common unit or scale. One benchmark might report accuracy, another F1, and another a BLEU score. You cannot simply add these numbers together and expect the result to mean something. Scores must be handled so they can be compared before they are combined.

Measure	What it reports	Example of why it resists direct addition
Accuracy	A percentage of correct answers	Higher is better, on a percent scale
F1	A balance of precision and recall	Not the same quantity as accuracy
BLEU	A text-overlap score used for generated text	Its scale is unrelated to accuracy
Pass rate	The share of tasks solved	Counts outcomes rather than answer matches
Latency	Time taken to respond	Lower is better, the opposite direction
Cost	Money spent per use	Lower is better, measured in currency

Scores in different units must be placed on a common scale before they can be compared or combined. The units shown here are the kinds the source and the course mention; the last two rows are generated examples of practical measures.

Weighted Totals, Step by Step

The source's guidance is to decide how important each benchmark is to your application and weigh the scores accordingly. Weighting makes your priorities explicit: a benchmark central to your product counts for more than one that is only loosely related. The pipeline below shows the flow from individual benchmark scores to one weighted ranking.

The following example is an invented illustration of the weighting idea, not a figure from the source. Suppose your application cares more about accuracy than about F1. Both scores below are already expressed on the same zero-to-one scale, which matters because real benchmark scores often are not on a shared scale and must be handled first.

Formula

Formula, step by step

1
S_m is the weighted total for model m, the single number used for ranking.
2
s_{m,i} is model m's score on benchmark i, already on a common scale.
3
w_i is the weight for benchmark i, reflecting how important that benchmark is to your application.
4
The sum runs over all n benchmarks, so each score contributes in proportion to its weight.
5
In this example the weights add up to one, so the total stays on the same zero-to-one scale as the inputs.

Worked example

Tracing the ranking as the weights change

Model A scores 0.80 on accuracy and 0.70 on F1. Model B scores 0.72 on accuracy and 0.90 on F1. Which model ranks first under two different weightings?

1
Set the first weights
Weight accuracy at 0.6 and F1 at 0.4. The weights sum to one.
2
Compute Model A's first total
0.6 times 0.80 is 0.48, and 0.4 times 0.70 is 0.28. The weighted total is 0.76.
3
Compute Model B's first total
0.6 times 0.72 is 0.432, and 0.4 times 0.90 is 0.36. The weighted total is 0.792.
4
Compare under the first weights
Model B has the higher total, 0.792 against 0.76, so Model B ranks first.
5
Change the weights
Now weight accuracy at 0.8 and F1 at 0.2. The weights still sum to one, but the priorities have shifted toward accuracy.
6
Recompute both totals
Model A becomes 0.8 times 0.80 plus 0.2 times 0.70, which is 0.78. Model B becomes 0.8 times 0.72 plus 0.2 times 0.90, which is 0.756.
7
Compare under the second weights
Model A now has the higher total, so Model A ranks first. The same scores produced a different winner because the weights changed.

Result

With accuracy weighted at 0.6, Model B ranks first. With accuracy weighted at 0.8, Model A ranks first. The raw scores did not change; only the stated priorities did.

What do you think happens?

Using the same two models, which one ranks first when accuracy is weighted at 0.8 and F1 at 0.2?

Model A
Model B

Reveal answer

Answer:

Model A ranks first, with a total of 0.78 against 0.756 for Model B.

Accuracy carries most of the weight, and Model A's higher accuracy of 0.80 outweighs Model B's stronger F1 of 0.90 once the weights shift.

Mistakes When Reading Leaderboards

Common mistake
Treating the overall rank as the answer for your application
What it looks like
Choosing the top model on a general leaderboard without checking the tasks your product performs.
Why it fails
A general ranking averages across many kinds of tasks, so it may not reflect the one or two capabilities your application depends on.
Do this instead
Match benchmarks to your application's capabilities and rank models on those.
Common mistake
Trusting a benchmark because of its name
What it looks like
Adopting a benchmark with an impressive title without checking what it measures.
Why it fails
Anyone can publish a benchmark, and many may not measure what their names suggest.
Do this instead
Ask what the benchmark actually measures and whether that matches the behavior you care about.
Common mistake
Assuming an old score still describes the current model
What it looks like
Using a rank recorded last year as if it were current.
Why it fails
Scores drift over time, and the source reports significant changes for GPT-3.5 and GPT-4 within a few months.
Do this instead
Treat dated scores as provisional and re-measure when the model changes.
Common mistake
Adding scores with different units directly
What it looks like
Summing an accuracy percentage and a BLEU score to get one total.
Why it fails
The scores do not share a common unit or scale, so the sum does not mean anything.
Do this instead
Put the scores on a common scale, then combine them with weights.
Common mistake
Ignoring the weights when reporting a ranking
What it looks like
Reporting a single winner without saying which priorities produced it.
Why it fails
As the worked example shows, changing the weights can change the winner from the same raw scores.
Do this instead
State your weights openly so others can see which priorities produced the ranking.

Building Your Own Shortlist

Public benchmarks have a limited job. Their purpose is to help you select a small subset of models that deserve more rigorous experiments. Those experiments should use your own benchmarks and metrics, because public benchmarks are unlikely to represent your application's needs perfectly and are also likely to be contaminated. The next topic in this course addresses contamination; here it is only introduced.

MEDIUM

Your product summarizes support tickets and classifies their urgency. List two capabilities it needs. For each, name the kind of benchmark you would look for, and note any capability you could not match to a benchmark. Then assign weights that sum to one and explain which model you would shortlist if one model were strong on classification and another were strong on summarization.

Hints

Start from the work the product performs, not from the names of leaderboards.
A capability with no matching benchmark is a gap that you must evaluate yourself.
Weights express priorities; a benchmark central to the product should carry the larger weight.

Your attempt (not saved)

Key Takeaways

A top rank on a general public leaderboard does not guarantee the best model for your application, because a general ranking averages across many kinds of tasks.
Start by matching benchmarks to the capabilities your application needs, and check that each benchmark is reliable, since anyone can publish one.
Scores drift over time, scores may be missing, and evaluations are costly, so you may need to run your own evaluations.
Scores in different units must be put on a common scale, then weighted by importance before they are combined into one ranking.
Public benchmarks serve to shortlist models for more rigorous evaluation using your own benchmarks and metrics.

More concepts starting with P