Understanding Comparative Evaluation
Ranking models is often the real goal, and two methods can produce that ranking: pointwise and comparative evaluation.
Why the Ranking Is the Real Goal
When you evaluate several models, the raw numbers are rarely the end goal. What you usually want to know is which model is the best fit for your purpose. In other words, the output you need is an ordering of the models from strongest to weakest. Two broad strategies can produce that ordering: pointwise evaluation and comparative evaluation. They reach the same kind of answer by different routes, and understanding those routes is the purpose of this article.
Both methods end in the same place: an ordering of the models. They differ only in what they measure along the way. Pointwise evaluation measures each model on its own and produces a score. Comparative evaluation measures models against each other and produces comparison results.
Three Poems, One Question
Imagine three models, called A, B, and C, that each write a short poem in response to the same prompt. You want to rank them from best to worst. Before reading on, think about how you would do it. Would you read each poem and give it a mark, or would you read the poems side by side and decide which one wins each time? Both options are reasonable, and the rest of this article follows each of them through to a final ranking.
Tracing a Pointwise Ranking
The defining feature of pointwise evaluation is independence. Each model is examined alone, and it receives its own score. Nothing about one model's result affects the score given to another. Once every model has a score, you sort the scores and read off the ranking. The highest-scoring model comes first.
The source illustrates this with a dance contest. To find the best dancer, you watch each dancer individually, give each one a score, and pick the dancer with the highest score. The method is simple to describe, and that simplicity is its appeal when the scores are easy to assign.
Ranking the three poems pointwise
Each poem is scored alone on a scale the evaluator chooses. The scores below are invented for this illustration, not drawn from any real evaluation. Produce a ranking.
- 1Score A alone
Poem A receives a score of 7. Its score does not depend on how B or C was written.
- 2Score B alone
Poem B receives a score of 9, again without reference to the other poems.
- 3Score C alone
Poem C receives a score of 6.
- 4Sort by score
Arranged from highest to lowest, the scores are 9 (B), 7 (A), 6 (C).
The pointwise ranking is B, then A, then C.
Tracing a Comparative Ranking
Comparative evaluation changes the unit of information. Instead of asking how good each model is in isolation, it asks which model wins when two are set against each other. The unit of information is a comparison result, not a score. The ranking is then computed from many such results.
The source gives the dance contest version of this idea. All candidates dance side by side, the judges say which dancing they prefer, and the dancer preferred by most judges is chosen. No single dancer receives a number; each judgment is a contrast between two performances.
Ranking the three poems head-to-head
Each pair of poems is put side by side and a judge says which one is preferred. The three results below are invented for this illustration. Produce a ranking from them.
- 1Compare A and B
Judges prefer poem B to poem A. B records a win and A records a loss.
- 2Compare A and C
Judges prefer poem A to poem C. A records a win and C records a loss.
- 3Compare B and C
Judges prefer poem B to poem C. B records a win and C records a loss.
- 4Count wins
B has two wins, A has one, and C has none. Ordering by wins gives B, then A, then C.
The comparative ranking is B, then A, then C, the same ordering the pointwise route produced, but reached from contests rather than from individual marks.
What do you think happens?
Suppose a judge had preferred C over A, while the other two results stayed the same. Would the ranking of B, A, and C still follow from counting wins?
Reveal answer
No. If C beat A, then A would have one loss against C and one win against B, and C would have one win against A. Counting wins would give B two wins, A one win, and C one win, so the top position would still be B, but A and C would now be tied on wins and the order between them would need a tie-breaking rule.
This shows that the ranking depends on the full set of comparison results, not just the single result you happen to look at. A single contest can change how the others are read.
Where the Two Routes Diverge
Both routes end in an ordering, so the difference lies in the middle steps. The pointwise route stops at a score for each model. The comparative route stops at a set of pairwise results. In both cases, the final step is the ranking itself. The diagram below places the two routes side by side so you can compare them column by column.
| Aspect | Pointwise evaluation | Comparative evaluation |
|---|---|---|
| What is measured | Each model's own score | Which model wins in a head-to-head contest |
| Unit of information | A score | A comparison result |
| Are models judged together | No, each is examined alone | Yes, models are set against each other |
| How the ranking is formed | Sort the scores | Compute the ranking from many comparison results |
| Typical fit for subjective responses | Can be challenging | Typically easier |
Two routes to the same kind of answer, an ordering of the models
Choosing a Route for Subjective Responses
The source makes a specific claim about subjective responses: pointwise evaluation can be challenging for them, while comparative evaluation is typically easier. The reason lies in the source's observation that judging a direct contrast is often easier than assigning an absolute number. The source's example is that it is easier to tell which of two songs is better than to give each song a concrete score. The flowchart below turns this reasoning into a sequence of decisions.
Real-World Settings for Comparative Evaluation
The source reports that comparative evaluation has been used in practice in several settings. Each one turns preferences into a ranking, though the details of each setting are not described in the source beyond what is listed here.
- In 2021, Anthropic used comparative evaluation to rank different models, which the source identifies as its first use in AI.
- LMSYS's Chatbot Arena ranks models on a leaderboard using scores computed from pairwise comparisons contributed by the community.
- Many model providers use comparative evaluation to assess their models in production.
- ChatGPT occasionally asks users to compare two outputs side by side. The outputs may come from different models or from the same model using different sampling variables, and the user's preference becomes a comparison result that can feed into a ranking.
Mistakes When Choosing an Approach
Common mistake Treating the scores from a pointwise run as if they were comparisons
What it looks likePoem B scored 9 and poem A scored 7, so someone concludes that B is nine times better than A.
Why it failsPointwise scores are sorted to produce an ordering. The numbers are not measurements of how much better one model is than another.
Do this insteadRead the pointwise output as an ordering. Do not read gaps between scores as exact amounts of quality.
Common mistake Assuming one head-to-head result settles the ranking
What it looks likeA single judgment preferring C over A is used to place C above A, ignoring the other results.
Why it failsThe ranking is computed from many comparison results together, so one contest is only part of the evidence.
Do this insteadGather the full set of pairwise results and compute the ranking from all of them.
Common mistake Choosing pointwise evaluation for subjective outputs without considering how stable the scores would be
What it looks likeA team scores creative poems on an absolute scale and assumes the scores mean the same thing across reviewers.
Why it failsThe source says pointwise evaluation can be challenging for subjective responses, and comparative evaluation is typically easier there.
Do this insteadAsk whether a direct contrast is easier to judge than an absolute number, and consider comparative evaluation when it is.
Practice: Ranking Four Answers
Four models, W, X, Y, and Z, each answer the same open-ended question. You are not given scores. Instead, you have these head-to-head results: W beats X, W beats Y, X beats Z, Y beats X, and Y beats Z. Use the results to produce an ordering from strongest to weakest, and explain which route you used and why it suited this task.
Hints
- Count how many contests each model wins across all of its comparisons.
- Check whether the top model is consistent with every result it appears in.
- Consider whether a pointwise route could have been used here, and what it would need from each model.
Key Takeaways
- Pointwise evaluation scores each model independently, sorts the scores, and reads the ranking from the sorted list.
- Comparative evaluation sets models against each other and computes a ranking from many head-to-head comparison results.
- The source states that pointwise evaluation can be challenging for subjective responses, while comparative evaluation is typically easier, because judging a direct contrast is often easier than assigning an absolute number.
- Comparative evaluation is used in practice, including Anthropic's 2021 work, LMSYS's Chatbot Arena, and comparisons that model providers run in production.