Preference Data and Model Alignment

When an AI model produces an answer, someone has to decide whether that answer is good. One option is to ask a general-purpose AI model to act as the judge. This is flexible, because the same model can be asked about many kinds of responses. It also has limits. A general model asked to judge might be slow or costly, and its verdicts can be hard to reproduce. Specialized AI judges address this by being models built or trained for one narrow evaluation job.

This article looks at three families of specialized judges: reward models, reference-based judges, and preference models. For each one, the useful questions are the same. What goes in? What comes out? And what does the output tell you about a response? Answering those three questions for each family is the fastest way to learn when each one is the right tool.

The defining feature of a reward model is that it looks at one response on its own, in the context of the prompt. It does not need a second answer to compare against, and it does not need a model answer written in advance. The input is the prompt together with a single response. The output is a score that expresses how good that response is for that prompt.

Reward models have been used in reinforcement learning from human feedback (RLHF) for many years. This is where the scores matter most: they can guide how a model is tuned.

Cappy, developed by Google in 2023, is a concrete case. Given a prompt and a response, Cappy produces a score between 0 and 1 that indicates how correct the response is. It is a lightweight scorer with 360 million parameters, which is much smaller than general-purpose foundation models. Its small size is part of its appeal: a compact scorer can be run many times over cheaply when a large batch of responses needs to be checked.

The reference-based judge changes what the score is measured against. Instead of asking how good a response seems in the abstract, it asks how well the generated response lines up with trusted reference answers. The reference answers are supplied as part of the input. Depending on the judge, the output is either a similarity score, which says how close the generated response is to the references, or a quality score, which says how good the generated response is compared to them.

BLEURT is the first published example. It takes in a (candidate response, reference response) pair and outputs a similarity score between the two. The score measures closeness to the reference, so the reference defines what counts as a match.

Prometheus is the second example, and it takes in four things: the prompt, the generated response, a reference response, and a scoring rubric. It outputs a quality score between 1 and 5, and it is set up so that the reference response is taken to earn a 5. The rubric tells the judge what good means for this task, and the reference anchors the top of the scale.

A preference model works on a different question from the first two judges. It does not assign each response its own number. It takes a prompt and two candidate responses, and it returns a decision about which of the two people would prefer. Its output is a relative choice rather than an absolute grade.

The target of a preference model is human preference, which is why people who study it often describe it as one of the most promising directions for specialized judges. If a model can predict what humans prefer, it can stand in for some of the human judgment that evaluation normally requires.

The reason this matters is cost. Preference data is essential for aligning AI models with human preferences, but it is challenging and expensive to collect. A preference model that predicts human choices well can make evaluation easier, and it can help make models safer to use, because its judgments follow the preferences that real people express. Two named examples are PandaLM, developed with work by Wang and colleagues in 2023, and JudgeLM, by Zhu and colleagues in 2023.

PandaLM goes a step beyond a bare verdict. Its output names the better response and also gives an explanation of its rationale. The explanation is useful to a reader who wants to check whether the verdict makes sense, not only whether the verdict matches what they expected.

The reward model and the preference model look most alike, because both can rate or rank responses to a prompt. The difference lies in what each one consumes and what it returns. The side-by-side view below makes that distinction explicit.

Choosing the right specialized judge depends on what you need from the evaluation. If you need a single score for one response, a reward model fits. If you need a score anchored to reference answers, a reference-based judge fits. If you need to know which of two responses people would prefer, a preference model fits. The map below shows how those task types connect to the judges that serve them.