Independent Evaluation Versus Comparative Evaluation

When people rank AI models by comparing them head to head, they often reason in a chain. If the first model is judged better than the second, and the second is judged better than the third, it feels obvious that the first is better than the third. This feeling is one half of a common misconception in comparative evaluation. The other half is the belief that this chaining always works for AI models. This article starts from that belief, tests it, and shows what changes when it fails and why the cost of comparing models rises as more of them enter the picture.

Transitivity is the property that lets a ranking algorithm skip some matches. Suppose three models have been compared in two of the three possible pairings. The results of those two comparisons can be combined into a third conclusion without running the third match. The algorithm observes two edges in a comparison graph and then adds a third edge by reasoning. Follow the steps below to see exactly what is measured and what is deduced.

The inferred edge is the one that carries risk. The two measured results are real observations. The A versus C result is not observed at all in this chain. If transitivity fails for these models, the inferred edge can point the wrong way, and nothing in the chain itself reveals the error, because the match between A and C was never run.

Chaining fails most visibly when the win relations form a loop. Imagine that A beats B, B beats C, and C beats A. Each result is a single observed outcome, yet the three together describe no consistent order. Any ranking that places A above B and B above C must place C below A, which contradicts the observed C over A result. This pattern is called an intransitive cycle, and the diagram below shows it as a loop of three edges.

Several papers that analyze Elo for AI evaluation list transitivity as a limitation of their methods. Their argument is that human preference is not necessarily transitive. A person can prefer one answer to a second, the second to a third, and still prefer the third to the first, depending on the situation. If human judgments behave this way, a ranking built on chaining those judgments can contain contradictions.

Non-transitivity can also come from the evaluation setup itself. Different model pairs are often judged by different evaluators and on different prompts. If the comparison of A against B used one set of prompts and evaluators, and the comparison of B against C used another, the two results were not measured under the same conditions. Chaining them mixes judgments that were never made on a common basis.

The first scalability problem is about how much data comparative evaluation needs. Comparative evaluation is data-intensive, and the number of model pairs to compare grows quadratically with the number of models. Each new model does not add one comparison. It adds a comparison against every model already in the set, so the total work grows much faster than the model count.

The pattern in the table follows from counting pairs. Each pair of distinct models needs one match, and the number of ways to choose two models from n is $\frac{n(n-1)}{2}$. Going from 5 models to 10 roughly quadruples the scores under independent evaluation would only double, but the pairwise matches jump from 10 to 45. The diagram below contrasts the two growth patterns directly.

Under independent evaluation, each model is scored once, so the cost of adding a model is one more evaluation. Under comparative evaluation, the cost of adding a model is one match against every existing model. This is why the data cost rises faster than the model count itself.

Transitivity is one way to reduce cost. A second way is to choose matches more carefully. So far, models have been selected randomly for each match, so every model pair appears in roughly the same number of matches. The source points out that not all model pairs need to be compared equally. Some pairs produce outcomes that are already clear, and spending more matches on them adds little information.

The key idea is that once the outcome for a model pair is known with enough confidence, that pair can be removed from further matching. An efficient matching algorithm should sample the matches that reduce the most uncertainty in the overall ranking. The loop below sketches one way to think about this process. It illustrates the stated principle and is not a specific algorithm from the source.

Notice what the loop does not do. It does not simply drop uncertain models or treat the ranking as final after a fixed number of matches. A pair leaves the loop only when its outcome is clear, and the overall ranking is revised after every new result. Cost falls because clear pairs stop consuming matches, while the ranking keeps its uncertainty estimates instead of hiding them.

A third bottleneck appears when a new model arrives. Under independent evaluation, only the new model needs to be evaluated, and it can be scored on a fixed benchmark without touching the scores of other models. Under comparative evaluation, the new model must be matched against existing models, and those matches can change the ranking of models that were already on the leaderboard. Adding a model is therefore not a local update. The whole ranking must be re-examined.

The same structure makes private models hard to evaluate. Imagine a company that has built a model on internal data and wants to know whether a public model would serve it better. Under comparative evaluation, the company would likely need to collect its own comparative signals and build its own leaderboard, or pay one of the public leaderboards to run a private evaluation. Both options reflect the same underlying cost: a comparison signal has to exist for the new model against the models it is being ranked with.