Elo Ratings for AI Model Evaluation

Comparative evaluation ranks AI models by having them compete head to head. Each match produces a result: one model is judged better than the other. A natural instinct is to treat these results as a chain of simple facts. If one model beats a second, and that second beats a third, it seems obvious that the first beats the third. This instinct is common, and it is only half of the story. The other half is the belief that chaining always works for AI models. This section sets up that belief so the rest of the article can test it.

Comparative evaluation is data-intensive, and the amount of data it needs grows quadratically with the number of models. The key detail is that a new model does not add a single comparison. It adds a comparison against every model already in the set. So each newcomer brings a fresh batch of matches, and that batch is larger than the last one. The total work therefore grows much faster than the model count itself.

Here $n$ is the number of models and $P(n)$ is the number of distinct pairs that could be compared. The expression counts each unordered pair once. The pattern is the reason the cost rises faster than the model count: the numerator contains $n^{2}$, so doubling the number of models roughly quadruples the number of pairs.

Transitivity is the reason an algorithm does not have to run every pair. Suppose three models have been compared in two of the three possible pairings. The results of those two measured comparisons can be combined into a third conclusion without running the third match. The algorithm skips the match and records an inferred result instead.

In this picture the two solid links are real observations, each made under its own conditions. The A to C link is different. It is a logical consequence of assuming transitivity, and no match between A and C was ever run to confirm it. The inference is only as trustworthy as the assumption behind it.

Several analyses of Elo for AI evaluation list transitivity as a limitation of their methods. Their concern is that the assumption may not match the data. Two separate mechanisms can produce non-transitive results, and they are worth distinguishing because they have different causes.

Transitivity is one way to reduce comparison cost. A second way is to choose matches more carefully. Up to this point, models have been selected randomly for each match, so every model pair appears in roughly the same number of matches. That is wasteful. Not all pairs need to be compared equally. Some pairs produce outcomes that are already clear, and spending more matches on them adds little information about the overall ranking.

The central idea is that once the outcome for a model pair is known with enough confidence, that pair can be removed from further matching. A more efficient matching algorithm samples the matches that reduce the most uncertainty in the overall ranking, and it stops pairs whose outcomes are already settled. The loop below illustrates this principle. It is a sketch of the stated idea, not a specific algorithm from the source.

A third bottleneck appears when a new model arrives. Under independent evaluation, only the new model needs to be evaluated. Under comparative evaluation, the new model must be matched against existing models, and those matches can change the ranking of models that were already on the leaderboard. Adding a model is therefore not a local update. The whole ranking may need to be re-examined.

The same structure makes private models hard to evaluate. Consider a company that has built a model on internal data and wants to know whether a public model would serve it better. Under comparative evaluation, the company would likely need to collect its own comparative signals and build its own leaderboard, or pay one of the public leaderboards to run a private evaluation. Both options reflect the same underlying cost: a comparison signal has to exist for the new model against the models it is being ranked with.