Building a Custom Model Leaderboard

When you need a language model for a product, you are rarely starting from zero. Many models have already been run against public benchmarks, and their scores are published. Those published results let you narrow a long list of candidates to a shortlist of models that look promising for your situation. This article follows the path from a pile of individual benchmark scores to an ordered ranking, which is the kind of ranking a leaderboard presents.

Keep one idea in mind throughout: a benchmark score is only one piece of evidence. To turn evidence into a decision, you have to decide which pieces to look at and how to weigh them together. Those two decisions, selection and aggregation, are what a leaderboard design actually consists of.

Benchmark results help you identify promising models, but there are too many benchmarks to examine them all. Selection is therefore required. A team cannot read through every available benchmark, check each model's result on each one, and weigh all of it. The practical question becomes which benchmarks deserve attention.

The answer is to start from the use case rather than from the benchmark list. Reading a list of benchmarks and asking which ones look interesting leads to a set chosen for convenience. Starting from the application leads to a set chosen for relevance. If your application writes code, coding benchmarks are the natural center of your evaluation. If it must also avoid producing harmful text, a benchmark that measures toxicity becomes relevant too.

Consider a hypothetical team building a product that writes code and must also keep its output free of harmful text. The team faces a large pool of available benchmarks. Its use case gives it two relevant directions: a coding benchmark that matches the main task, and a toxicity benchmark that matches the safety requirement. Other benchmarks in the pool may be well known, but they measure tasks the product never performs, so they drop out of the evaluation.

The diagram shows the filter step. The use case acts as the test each benchmark must pass. Only the benchmarks that reflect the tasks and behaviors the application needs move on to the next stage. This filter is where your judgment first enters the process, and the rest of the leaderboard can only be as meaningful as the set that survives it.

After selection, each selected benchmark produces results for every model. Those results still sit in separate piles, one per benchmark. Benchmark aggregation combines results across benchmarks to rank models. The output of aggregation is the leaderboard: the ordered list the team actually reads.

The difficulty is that a model can win on one benchmark and lose on another. Consider a model that does better on a coding benchmark but worse on a toxicity benchmark. Aggregation has to resolve that tension, because a ranking must place each model somewhere. There is no single fixed rule that settles every such case. There are multiple ways to aggregate benchmark results, so the method you choose is a real decision, and it is part of the evaluation rather than a neutral step.

Read the diagram as a pipeline. Candidate benchmarks are narrowed to the selected set, each selected benchmark produces results for every model, and the aggregation method turns those per-benchmark results into the order the leaderboard displays. Your judgment enters at two points: the filter step, where benchmarks are chosen, and the combine step, where the aggregation method is chosen.

Every leaderboard design answers two questions. The first is which benchmarks to include. The second is how to aggregate their results. The two are linked: the benchmarks define the evidence, and the aggregation decides how that evidence becomes an order. Changing either one can change where a model appears on the final list.

Treat these as two separate design decisions, each with its own justification. Selection should follow from the use case. Aggregation should follow from the priorities you bring to the ranking, because the method determines how a win on one benchmark is weighed against a loss on another.

This example shows why the method is part of the evaluation. If the team had not made the aggregation choice explicit, the order would look objective while hiding a judgment about which failure matters more.

Public leaderboards already demonstrate how the two decisions can be made, so studying how they are built gives you a reference point. You do not have to design the process from nothing. You can borrow the logic of an existing public leaderboard, adapt its benchmark list to your use case, and adjust its aggregation to match your priorities.