AI Judges in Model Evaluation
The Elo rating algorithm is a rating algorithm used to compare the performance of different models or entities.
Why Models Need One Comparable Number
When people compare AI models, they often want a single number that places each model on a scale. That makes it possible to say which model is ahead and by how much, without reading every individual result. The Elo rating algorithm is one method that produces this kind of rating. It is used to compare the performance of different models or entities.
This article looks at what that algorithm is for, and at a case where the name attached to the ratings became separated from the method that created them. The case is useful because it shows that a label on a number is not the same thing as a record of how the number was made.
Method Versus Output
Two ideas are easy to blur together here. The first is the method: the algorithm that turns comparisons into ratings. The second is the output: the ratings themselves. The name Elo belongs to the method. A rating is only an output.
The source also explains why this distinction matters. In some cases evaluation can take up the majority of the budget, even more than response generation. A team that spends heavily on evaluation has a strong interest in knowing which method produced each number it publishes. A rating that carries an old name can make it harder for readers to judge what the numbers mean.
Where the Name Stops Tracking the Method
A rating name such as Elo score describes an output, and that output can keep its name after the method that produced it has been dropped. Chatbot Arena is the case the source uses to show this. Chatbot Arena stopped using the Elo rating algorithm, but its developers continued referring to their ratings as Elo scores for a while.
The diagram below traces the relationships in this case. The Elo rating algorithm is the method that no longer produces the Chatbot Arena ratings. The Chatbot Arena ratings are the output. The label Elo score is the name that stayed attached to that output. The source also says the Chatbot Arena ratings were scaled from a Bradley-Terry model, which is a separate model named in the source.
The diagram shows where the name stops tracking the method. The label still points back to the Elo algorithm, but the ratings that carry the label no longer come from that algorithm. The name survived because it describes the output, and outputs keep their labels after the process that made them has changed.
Mistakes About Elo Names
Common mistake Assuming that a score called an Elo score must come from the Elo rating algorithm
What it looks likeA set of model ratings is labeled Elo scores, so a reader concludes the Elo algorithm must have produced them.
Why it failsThe name describes an output. The source shows that an output can keep its name after the method that produced it has been dropped, as happened with Chatbot Arena.
Do this insteadTreat the label as a claim to check. Ask which method produced the numbers, and look for that information in the source rather than relying on the name.
Common mistake Treating the name Elo as a description of how the ratings were computed
What it looks likeA reader assumes Chatbot Arena's ratings were computed by the Elo update procedure because the ratings are called Elo scores.
Why it failsThe name belongs to the method, and the source says Chatbot Arena stopped using the Elo rating algorithm.
Do this insteadSeparate the method from the output and state which one the evidence supports.
Common mistake Filling in the missing computation details from general knowledge
What it looks likeA reader assumes the Chatbot Arena scaling works the same way as the scaling in some other rating system.
Why it failsThe source says the ratings were scaled from a Bradley-Terry model but does not explain that scaling, so any detail beyond that is not established by the source.
Do this insteadReport only what the source states, and mark the rest as not covered.
Tracing a Claim About Chatbot Arena
Checking whether the Elo name proves the method
A blog post states: Chatbot Arena ratings are Elo scores, so they were computed with the Elo rating algorithm. Using only the source, decide whether this claim is supported.
- 1Identify the claim's two parts
The claim links a name, Elo score, to a method, the Elo rating algorithm. It treats the name as proof of the method.
- 2Check what the source says about the method
The source states that Chatbot Arena stopped using the Elo rating algorithm. So the method that the claim relies on is stated to be no longer in use for these ratings.
- 3Check what the name is attached to
The source says developers continued referring to their ratings as Elo scores for a while. The name is therefore an output label that persisted, not evidence of the procedure.
- 4Check the scaling detail
The source says the ratings were scaled from a Bradley-Terry model but does not explain that scaling. Nothing in the source supports filling that gap with the Elo update procedure.
- 5Decide
The claim is not supported. The name alone does not establish the method, and the source points the other way on the method used.
The claim that the ratings were computed with the Elo rating algorithm is not supported by the source. The name Elo score is consistent with a name that outlived the method.
What the Source Does and Does Not Establish
It is important to be precise about the limits of the evidence. The source supports several statements about Chatbot Arena, but it leaves the computation largely unexplained. The comparison below separates what is stated from what is not.
Testing Your Reading of Ratings
What do you think happens?
Must a rating called an Elo score come from the Elo rating algorithm?
Reveal answer
No, the name describes an output that can outlive its method
The source shows that Chatbot Arena stopped using the Elo rating algorithm but its developers kept calling their ratings Elo scores for a while. A name describes an output, so it cannot by itself certify the method that produced it.
A leaderboard lists model ratings under the heading Elo scores. Write two sentences: one stating what you can conclude from the heading alone, and one stating what you would need to check in the source before concluding how the ratings were produced.
Hints
- Recall the difference between the method and the output.
- The source says the Elo name outlived the Elo method in one case, so the heading alone cannot settle the question.
- The source does not explain the Chatbot Arena scaling, so say so if the question depends on it.
Key Takeaways
- The Elo rating algorithm is a rating algorithm used to compare the performance of different models or entities.
- A rating name such as Elo score describes an output, and the output can keep its name after the method that produced it is dropped.
- Chatbot Arena stopped using the Elo rating algorithm, but its developers continued calling their ratings Elo scores for a while, so a score with that name does not by itself prove the Elo algorithm produced it.
- The source says the Chatbot Arena ratings were scaled from a Bradley-Terry model but does not explain that scaling, so details of how those ratings were computed remain unverified.