Limitations of AI Judges

Imagine a team that has built two chatbots and wants to know which one gives better answers. Reading thousands of responses by hand is slow, so the team hands the job to an AI judge: a model that reads two answers to the same question and says which one is better. The judge returns a verdict in seconds, and the team starts making decisions based on it.

The catch is that an AI judge does not see responses from a neutral place. It can lean toward certain answers for reasons unrelated to quality. Human graders have their own preferences too, but the particular leanings of AI judges are different and worth learning to spot.

Every grader has leanings, and AI judges are no exception. Different AI judges also carry different leanings, so a bias present in one judge may be absent in another. The practical consequence is that a score is not just a number. It is a number produced by a particular judge, and that judge's tendencies shape it.

Self-bias is the tendency of a judge to favor responses produced by its own model. The source connects this to how a model produces text in the first place. A model generates responses by estimating which continuation is most likely. When the same model later evaluates a response, the same mechanism tends to give high marks to the response it would have produced itself. The judge recognizes the style and choices it already finds natural, and it rewards them.

The effect has been measured. In an experiment by Zheng et al. in 2023, GPT-4 favored itself with a 10% higher win rate, and Claude-v1 favored itself with a 25% higher win rate. The numbers differ between models, which reinforces the point that judges do not all share the same leanings.

First-position bias means the judge tends to favor whichever answer is shown first. It matters most in pairwise comparison, where the judge reads two answers and picks one. If the judge leans toward the first answer, the order in which the answers are presented becomes part of the result. Two identical comparisons with the answers swapped could produce different verdicts, and the difference would come from the presentation, not from the content.

The source notes that first-position bias runs opposite to the human recency bias, which tends to favor the most recent item. Recognizing the direction of the leaning helps you know which test to run.

The source describes a practical remedy: repeat the same test several times with different orderings, or write the prompts more carefully. The most concrete version of the first idea is to run each pair in both orders. In the first run, Answer A sits in position 1 and Answer B in position 2. In the second run the positions are swapped. Each answer now appears once in each position, so position can no longer explain a consistent win.

Verbosity bias is the tendency of a judge to favor longer answers regardless of quality. The source notes that this bias can outweigh correctness, so length and factual content should be checked separately rather than assumed to move together.

The most striking finding comes from Wu and Aji in 2023. They found that both GPT-4 and Claude-1 preferred longer responses of about 100 words that contained factual errors over shorter responses of about 50 words that were correct. In that setting, length outweighed correctness.

Saito et al. in 2023 studied the same bias for creative tasks. Their result was that when the length difference is large, for example when one response is twice as long as the other, the judge almost always picks the longer one. The source treats this as a strong pattern, so a large gap in length should be read as a warning sign whenever the verdict depends mostly on it.

There is one encouraging detail. Both Zheng et al. (2023) and Saito et al. (2023) found that GPT-4 is less prone to verbosity bias than GPT-3.5, and the source suggests this bias might fade as models become stronger. That is a suggestion from a few studies rather than a guarantee, so a team should still test for the bias in the judge it actually uses.

The three biases do not act in isolation. A single pairwise verdict can be shaped by the position of each answer, by its length, and by whether the judge recognizes one answer as its own output. Each leaning is a separate pull on the same final decision. The verdict is what remains after all of those pulls are combined, and the verdict alone does not reveal which pull was strongest.

Biases are not the only limitation. The source notes that AI judges share the same limitations as all AI applications, including privacy and intellectual property concerns. These are not about how the judge scores answers. They concern what happens to the data around the scoring.

If the judge is a proprietary model, using it means sending your data to that model. That data may include the responses you are evaluating, which may themselves be sensitive or owned. If the model provider does not disclose its training data, you cannot know for certain whether the judge is commercially safe to use.