Ranking Algorithms for Model Comparison

Suppose a team wants to find out which of several language models gives answers that people prefer. Each time they gather feedback, a person reads a response to a prompt. The design of that feedback loop matters a great deal, because it determines what the evaluator can compare and what the final results can mean.

A common first impression is that comparative evaluation simply shows a user one model's answer, then another model's answer, and records which one the user liked more. That description sounds reasonable, but it describes a different method. Comparative evaluation has a specific structure, and confusing it with a neighboring technique leads to wrong conclusions about what the results mean.

The defining difference is in what the evaluator is shown. In A/B testing, one candidate is shown at a time, so the evaluator judges each candidate on its own. In comparative evaluation, two or more models answer the same request, and the evaluator sees them together and picks the winner of each match. The judgment is therefore relative: it says which response is preferred in that contest, not how good a response is in isolation.

Because the two methods ask different questions, they are not interchangeable. A single-candidate study can tell you how an evaluator responds to one answer. A comparative study tells you which of several answers wins head-to-head contests. Using the result of one method to make a claim that only the other can support is the mistake this distinction is meant to prevent.

Comparative evaluation records each contest as a row: which two models competed, and which one won. Each row is one judgment by an evaluator, and no row says anything about the models in isolation. Taken together, these rows form a history of head-to-head matches, which is the raw material for ranking.

A raw history is not yet an answer. A handful of matches cannot say whether one model is better than another in general. To answer that kind of question, the matches between the same pair of models must be summarized. That summary is the win rate of model A over model B, which estimates the probability that A is preferred over B.

The calculation is a simple count. Find every match between A and B, count how many A won, divide by the total number of those matches, and express the result as a percentage. Matches against other opponents do not enter the numerator or the denominator.

With two models, ranking is straightforward: the model that wins more often ranks higher. With five models, the problem changes. A table of ten pairwise win rates does not give an obvious order, because some models win against some opponents and lose to others. Models can have a favorable record against one rival and an unfavorable record against another, so no single pair decides the order.

This is the job of a rating algorithm. It first assigns each model a score based on the comparative signals, and then ranks the models by those scores. Each match contributes to the scores, and the accumulated scores produce the final ordering. The steps below trace that flow across a short match history.

Several rating algorithms have been adapted from other fields. Elo, Bradley-Terry, and TrueSkill come from sports and video games, where players have long been ranked from head-to-head results. Because these algorithms were built for different settings, they can treat the same comparison data differently.

Because different algorithms can order the same models differently, a ranking should not be accepted just because it was produced. A ranking is judged by how well it predicts future match outcomes. If a ranking places model A above model B, it should correctly anticipate which of them evaluators will prefer in matches not yet observed.

Comparative evaluation measures which response evaluators prefer in head-to-head contests. That is a precise measurement, but it answers only the question it was built for. A preference record says which answer was liked more in that contest. It does not, by itself, establish that the preferred answer is the more accurate, safer, or more complete one on every dimension a team might care about.