Using AI as a Judge

Suppose you run one AI-generated answer through two different evaluation tools, and both report a relevance score for it. It is natural to assume that the two numbers measure the same thing, so a higher number from one tool should mean the same kind of quality as a higher number from the other. This belief is common and understandable, and it is the belief this article corrects.

To see why it fails, we first need to understand what an AI judge is being asked to do. An AI judge is a model that you give some text and a question about that text, and it returns a verdict. The verdict might be a number on a scale, a True or False answer, or a choice between two candidates. What the judge can see, and what question it is answering, determines what kind of judgment comes back.

An AI can judge responses in three main ways. In the first, it evaluates a single response by itself, given the original question. In the second, it compares a generated response to reference data, such as a known good answer, and decides whether the two match. In the third, it compares one generated response against another and decides which is better, or predicts which one users will likely prefer.

The three approaches differ mainly in what the judge is given to look at. Scoring alone needs the question and the response. Reference checking needs the question, the response, and a reference answer. Pairwise comparison needs the question and two responses, and it does not need a reference answer at all, because the judge only has to see the two candidates side by side and pick one.

Scoring alone is the easiest mode to follow step by step, so start there. The judge still needs the original question, because a response can only be called good or bad relative to what was asked. The source's naive example prompt defines its scale directly inside the instruction: a score of 1 means very bad and a score of 5 means very good. The judge is then given the question and the answer and asked to fill in a score.

Every judge prompt, whatever tool produces it, has three parts you should be able to find: the judging question, the score scale, and the required return format. In the example above, the judging question is the instruction to rate the answer. The scale is written into that instruction as the range 1 to 5 with its two endpoints defined. The return format is the final line, which asks for only the number after Score.

The reference approach changes the question the judge answers. Instead of rating quality on a scale, the judge receives a reference answer, which is a response someone already considers correct, and it decides whether the generated answer is the same. The source describes this as an alternative to human-designed similarity measurements. Rather than writing a rule that measures how close two answers are, you hand that comparison to an AI judge. The output for this mode is True or False.

The pairwise approach compares two generated responses and decides which is better, or predicts which one users will likely prefer. The source names two uses. One is generating preference data for post-training alignment, which is discussed in Chapter 2 of the source. The other is ranking models using comparative evaluation, which the source covers in the following section. Because the judge only picks between two candidates, it produces a choice rather than an absolute score.

A general-purpose judge can also be asked to evaluate against criteria you choose. The source gives two examples. A developer building a roleplaying chatbot might ask whether a response is consistent with the role users want, such as whether it sounds like something Gandalf would say. A developer generating promotional product photos might ask for a rating from 1 to 5 of how trustworthy the product in the image looks. Each of these is a custom criterion written by the developer, not a standard one.

Some AI tools come with built-in criteria, which are predefined names for common things to judge. The source lists examples of these as offered by some tools as of September 2024. Azure AI Studio and MLflow.metrics both include a criterion called relevance, and Ragas includes one called answer relevance. Because the source notes that these tools change, treat any such list as a snapshot rather than a permanent catalogue.

The shared label suggests a shared meaning, but the source is explicit that AI judge criteria are not standardized. Its direct illustration is that Azure AI Studio's relevance scores might be very different from MLflow's relevance scores. The reason is that a score depends on the judge's underlying model and on the prompt used to ask the question. Two tools can use the same word for a criterion while asking the judge to do different work.