Ranking Models with Comparative Evaluation

Earlier in this chapter, a general-purpose AI model was used to evaluate the output of another model. That approach is flexible, because the same model can be asked about many kinds of responses. It also has limits. A general model asked to judge might be slow or costly, and its verdicts can be hard to reproduce. Specialized AI judges are models built or trained for one narrow evaluation job. This article follows three families of them: reward models, reference-based judges, and preference models.

Before reading on, try the question below. Each specialized judge is designed around a different kind of input, so the question is really about what information the judge needs in order to do its job.

A reward model is defined by one feature: it looks at a single response on its own, in the context of the prompt. It does not need a second answer to compare against, and it does not need a model answer written in advance. The input is the prompt together with a single response. The output is a score that expresses how good that response is for that prompt.

Reward models have been used in reinforcement learning from human feedback (RLHF) for many years. The scores they produce can guide how a model is tuned. Cappy, developed by Google in 2023, is a concrete case. Given a prompt and a response, Cappy produces a score between 0 and 1 that indicates how correct the response is. It is a lightweight scorer with 360 million parameters, which is much smaller than general-purpose foundation models.

The reference-based judge changes what the score is measured against. Instead of asking how good a response seems in the abstract, it asks how well the generated response lines up with trusted reference answers. The reference answers are supplied as part of the input. Depending on the judge, the output can be a similarity score, which says how close the generated response is to the references, or a quality score, which says how good the generated response is compared to them.

Two published judges show the two output styles. The walkthrough below treats each one separately, because they answer different questions about the generated response.

A preference model works on a different question from the first two judges. It does not assign each response its own number. It takes a prompt and two candidate responses, and it returns a decision about which of the two people would prefer. Its target is human preference. The people who study it often describe it as one of the most promising directions for specialized judges, because a model that can predict what humans prefer can stand in for some of the human judgment that evaluation normally requires.

The reason this matters is cost. Preference data is essential for aligning AI models with human preferences, but it is challenging and expensive to collect. A preference model that predicts human choices well can make evaluation easier, and it can help make models safer to use, because its judgments follow the preferences that real people express. Two named examples are PandaLM, developed with work by Wang and colleagues in 2023, and JudgeLM, by Zhu and colleagues in 2023.

PandaLM goes a step beyond a bare verdict. Its output names the better response and also gives an explanation of its rationale. The explanation is useful to a reader who wants to check whether the verdict makes sense, not only whether the verdict is the one they expected.

Choosing the right specialized judge depends on whether you need a single score, a score anchored to references, or a head-to-head preference. The decision path below turns those three needs into questions you can answer about your task.

Read the flowchart from the top. The first question separates the preference model from the other two, since only the preference model takes two responses and returns a choice. The second question separates the reference-based judge from the reward model, since only the reference-based judge uses reference answers to anchor its score.