Evaluating AI Outputs with LLM-as-Judge
LLM-as-judge works with caveats: GPT-4 class judges reach roughly 80 percent agreement with human raters, which is approximately the human-human level, but this is not a guarantee for every task.
Why a Judge Seems Neutral
When a team first builds an LLM-as-judge setup, a common assumption appears quickly. If a capable language model can read two answers and say which one is better, then its verdict should be a neutral measurement of quality. The judge seems to have no stake in the outcome, no favorite answer, and no reason to lean one way.
This belief is understandable, but it is wrong in an important way. LLM-judges do produce useful signals, and they can agree with human raters at a surprisingly high rate. However, they also carry predictable biases, and their scores only mean something after they have been checked against human judgment on the data you actually care about.
Before reading further, ask yourself a concrete question. Suppose you show a judge two answers and it picks the first one. What would you need to know before you believed that the first answer is genuinely better? Keep that question in mind, because the rest of this article is about the things that can make the verdict misleading even when it looks confident.
Tracing a Pairwise Verdict Through a Swap
A pairwise judge sees two candidate answers, usually labeled by slot, and returns a verdict. Consider a small thought experiment. Answer X and Answer Y respond to the same question. The judge is shown X in slot A and Y in slot B, and it prefers slot A. Now the same two answers are presented again with their slots swapped, so Y is in slot A and X is in slot B. A judge that tracks quality should now prefer X, which is the same answer it preferred before. A judge with position bias may keep choosing slot A, which means the winner changes even though the quality did not.
Slot A: Answer X Slot B: Answer Y Verdict: prefers slot A
Answer X wins because it occupied the first slot
Slot A: Answer Y Slot B: Answer X Verdict: prefers slot A
Answer Y wins because it now occupies the first slot
Read the diagram carefully. The winning answer changed from X to Y, but the rule the judge followed stayed the same: it kept favoring whatever sat in slot A. If the team reported that the judge preferred Y in this run, that result could be an artifact of layout rather than a real difference in quality. The judge's answer looked like a judgment, but in this case it behaved more like a reflex tied to presentation.
Three Named Judge Biases
LLM-judges have predictable biases. Because these biases are systematic rather than random, they push verdicts in consistent directions, and a team can learn to look for them. Three are named most often. Each one is a distinct source of error, and each sits underneath a judge score that may look objective.
Predicting How Each Bias Moves a Verdict
Once the three biases are named, you can ask a sharper question about any verdict: which direction would this bias push the result, and does that direction explain the outcome better than quality does? The table below organizes the prediction. Use it as a checklist when you review a surprising judge result.
| Bias | What the judge favors | Predicted verdict change | Check to run |
|---|---|---|---|
| Position bias | The first or last response in the presented pair | The winner flips when slots are swapped, even though quality did not change | Run the pair in both orders and compare verdicts |
| Verbosity bias | The longer response | The longer of two equally correct answers wins | Compare verdicts on equally correct pairs that differ only in length |
| Self-enhancement bias | Outputs that resemble the judge's own outputs | Answers from the judge's own model family score higher than comparable answers from other models | Compare ratings for the same answers judged by different model families |
Predicting how each named bias would change a judge's verdict
The Roughly 80 Percent Agreement Figure
The main reason this approach is taken seriously is its agreement with people. GPT-4 class judges reach roughly 80 percent agreement with human raters. This figure is often described as approximately the same level of agreement that humans reach with one another, which is why it is treated as a meaningful ceiling rather than a disappointing number.
The agreement can be high on average and still hide systematic errors on the specific kinds of cases your product produces. Those systematic errors are exactly the biases described above. An 80 percent average says little about whether the errors fall on the cases that matter most to your customers, which is why the next step is calibration.
Calibrating the Judge on Your Own Data
The central lesson is that an LLM-judge becomes useful only after calibration. Without it, the judge produces scores, the team acts on those scores, and the scores have an unknown relationship to the actual quality bar the customer cares about. A number can look precise and still measure something other than what you intended.
Calibration means adjusting the judge against human labels on your own data. It connects the judge's scale to human judgment on the examples where the judge will be used. Human labels and judge scores attach to the same evaluation data, and the calibration step compares them. Only after that comparison should the scores be trusted for decisions.
Read the flowchart from left to right. Human labels enter at the comparison step, alongside the judge's scores, and both attach to the same evaluation data. The judge is not calibrated once in general. It is checked against labels on the specific data where it will be used, so a judge that calibrates well on one kind of task still needs checking when the task changes.
Mistakes When Trusting a Judge
Common mistake Assuming the judge is unbiased because it has no stated preference
What it looks likeA team reads one run where the judge picked the first answer and concludes that the first answer is better.
Why it failsPosition bias produces verdicts that follow layout, so a single fixed-order run cannot separate quality from presentation.
Do this insteadRun the same pair in both orders and check whether the verdict follows the answer or the slot.
Common mistake Treating the 80 percent agreement figure as a guarantee
What it looks likeA team sees roughly 80 percent agreement with human raters and skips validation on its own product data.
Why it failsThe figure is an average under particular conditions, and it can hide systematic errors on the cases the product produces.
Do this insteadCalibrate against human labels on your own data before acting on the scores.
Common mistake Reading a longer answer's win as a quality win
What it looks likeA padded answer wins against a concise answer that is equally correct.
Why it failsVerbosity bias favors longer responses, so the win may reflect length rather than merit.
Do this insteadTest pairs of equal correctness that differ only in length and see whether the verdict tracks length.
Common mistake Acting on scores before any human labels exist for the evaluation data
Reading a Judge Result Critically
A judge compares two model outputs on your product's support questions. Across the full set, Model P wins most comparisons. Before you report that Model P is better, describe two checks you would run. Name the bias each check targets and state what result would change your conclusion.
Hints
- Think about what happens to the verdict when the two outputs swap slots.
- Consider whether Model P's answers are systematically longer, or whether the judge shares a model family with Model P.
- A good check is one whose result could contradict the original conclusion.
Key Takeaways
- An LLM-judge is not unbiased by default. Its verdicts can follow position, length, or resemblance to its own outputs rather than quality.
- GPT-4 class judges reach roughly 80 percent agreement with human raters, approximately the human-human level, but this is an average under conditions that must be verified and does not guarantee agreement on every task.
- Position bias, verbosity bias, and self-enhancement bias push verdicts in consistent, predictable directions, so each can be tested for.
- Calibrating the judge against human labels on your own data connects its scores to the real quality bar, and it should happen before the team acts on the scores.