Preference Models

Imagine you have built two chatbots and want to know which one is better. A natural plan is to hand each answer to a grader, collect a number for every answer, and then declare the bot with the higher average the winner. The plan feels objective because it produces numbers, and numbers seem to settle arguments.

The belief hiding inside that plan is that a score is a fixed yardstick: a 4 means the same thing no matter who assigned it, which judge was used, or when the measurement was taken. For open-ended AI output, that belief breaks down. This article explains why, and what evaluators do instead. Keep the two chatbots in mind as you read, because each idea below changes how you would judge that contest.

Automatic evaluation matters here because open-ended, powerful models are hard to evaluate, and the difficulty grows with capability. Human evaluation is always helpful for sanity checks and is essential in many cases. The material in this chapter concentrates on automatic approaches, because those are what allow evaluation to run at the scale modern applications require.

The automatic approaches fall into three families: functional correctness, similarity scores, and AI as a judge. The first two are exact. A response either matches a reference or passes a check, with little room for interpretation. AI as a judge is different. It is a subjective evaluation, which means the verdict depends on how a particular judge reads the response.

Because AI as a judge is subjective, its outputs depend heavily on the judge that produced them. A judge is not a neutral ruler. A score taken from one judge tells you little about a score taken from another, even when both are labeled as measuring the same quality. This is why evaluators say such scores must be interpreted within the context of the judge being used.

A second problem concerns time. AI judges, like any AI application, get iterated on, so their judgments change as they are improved. If the judge changes between one measurement and the next, a rise or fall in your application's score may reflect the judge rather than the application. That is why AI judges are considered unreliable as benchmarks for tracking an application's changes over time. A benchmark needs a stable reference point, and an iterating judge is not one.

Once you accept that individual scores are fragile, the question becomes how to rank several models fairly. There are two routes. The first is to evaluate each model independently, attach a score to each one, and then sort the models by those scores. The second is comparative. Rather than scoring a model in isolation, you ask which of two models is better. Each comparison yields a relative signal, and the collection of those signals is used to order the models.

Comparative evaluation ranks models based on relative performance, and the source notes that the practice is common in sports, especially chess, and is gaining traction in AI evaluation. The Elo rating algorithm is one method for turning comparative results into a ranking. Notice what the comparative route avoids: it never needs to claim that Model A earned a particular absolute number. It only needs to know which of two responses is preferred in a given comparison.

Comparative evaluation runs on preference signals, which record a judgment about which response is preferred. The source points out that preference signals are expensive to collect, and that this cost matters for two reasons: comparative evaluation needs them, and so does the post-training alignment process. A preference model is a specialized AI judge whose job is to predict which response users would prefer. Because collecting preference signals by hand is costly, the field developed preference models to stand in for some of that work.

The same caution applies to individual metrics. The BLEURT score is a metric for judging the quality of responses, and its range is roughly between -2.5 and 1.0. The source singles out this range as confusing. The range can be arbitrary, which illustrates the broader problem of criteria ambiguity with AI judges. A number with an unclear scale cannot be interpreted on its own, and a score that sits near the top of an unclear range is not automatically good. Scales such as a Likert scale can share the same weakness when the meaning of each point is not pinned down.

Rating labels carry a similar hazard. The source records that Chatbot Arena stopped using the Elo rating algorithm, yet its developers continued, for a while, to call their model ratings Elo scores. The ratings were derived from a Bradley-Terry model and then scaled. The lesson is that a label such as Elo score does not by itself tell you which method produced the number. When you read a leaderboard, check what produced the ratings rather than trusting the name on the column.

A general-purpose AI model can be asked to evaluate another model's output. That approach is flexible, because the same model can be asked about many kinds of responses. It also has limits. A general model asked to judge might be slow or costly, and its verdicts can be hard to reproduce. Specialized AI judges are models built or trained for one narrow evaluation job. The useful questions for each one are the same: what goes in, what comes out, and what the output is meant to tell you about a response.

The two judges that look most alike are the reward model and the preference model, because both can rate or rank responses to a prompt. The difference is in what they consume and what they return. A reward model consumes one prompt and one response and returns a score for that response. A preference model consumes one prompt and two responses and returns which of the two is preferred. The side-by-side diagram makes that distinction explicit.