Reward Models and Verifiers

A language model rarely has to produce only one answer. When you sample several candidate outputs for the same prompt, a practical question appears: which candidate should the system return? There are two broad options. One is to show several candidates to a person and let that person pick the one that suits them. The other is to build an automatic rule that ranks the candidates and keeps the top one. This article is about the second option, the scoring rules that let software choose.

Every rule covered here starts from the same raw material. Each token in a candidate has a probability that the model computed while generating it, so every candidate carries a trail of numbers. The methods differ in how they combine that trail into a single score. In one family of methods, they replace the trail with a separate scoring model. As you read, keep one question in mind: if two candidates have different lengths, will the scoring rule treat them fairly?

The most direct selection rule is to pick the output with the highest probability. An output is a sequence of tokens, and the probability of the whole sequence is built from the per-token probabilities. The rule is that the probability of an output equals the product of the probabilities of all the tokens in it. Each new token multiplies the running total by one more factor.

The later factors are conditional. The probability of the second token is a probability given the first token, and the probability of the third token is given the first two. The product therefore reflects the whole chain of choices that produced the sequence, not three independent guesses. Because every factor is a number between 0 and 1, each added token can only keep the total the same or make it smaller.

The diagram uses an invented three-token candidate whose token probabilities are 0.5, 0.4, and 0.5. The first token alone has probability 0.5. Adding the second token gives 0.5 times 0.4, which is 0.2. Adding the third gives 0.2 times 0.5, which is 0.1. The total went down at every step. This is why a longer output tends to have a smaller raw probability than a shorter one, even when each of its individual tokens is fairly likely.

Multiplying many small numbers is awkward in practice, so working with probabilities on a log scale is easier. The key fact is that the logarithm of a product equals the sum of the logarithms. Taking the log of a sequence probability therefore turns the chain of multiplications into a chain of additions. The log probability of a sequence is the sum of the log probabilities of its tokens.

The before column multiplies three factors to get 0.1. The after column takes the natural logarithm of each factor and adds the results, arriving at about -2.30. This is the same quantity, because the natural log of 0.1 is about -2.30. The benefit is practical: a running sum of log values is easier to compute and keeps numbers in a comfortable range, instead of shrinking toward zero with every multiplication. The source does not specify a logarithm base, and the choice of base does not change which candidate ranks highest, so the arithmetic in this article uses natural logarithms.

Once outputs are scored by summed log probability, longer outputs face a structural disadvantage. Each additional token adds another negative term, so a long candidate's total usually sits lower than a short candidate's total. If the selection rule simply took the highest sum, it would keep favoring brevity, even when a longer output is the better answer. The sum measures how many tokens there were as much as how likely each one was.

The fix is to normalize by length. Dividing the summed log probability by the number of tokens gives the average log probability. This measures how strongly each token contributed on average, rather than how many tokens there were. The selection method then picks the candidate with the highest average. The source reports that this is the approach the OpenAI API uses, as of its writing.

Probability-based rules depend only on the numbers the model already produced. A different family of methods uses a separate model, called a reward model, to score each candidate. The reward model examines each output and assigns it a score, and the system selects the output with the highest score. The source describes this as the same idea as the verifiers discussed in the previous section.

The source does not describe how the reward model is built internally. What matters for selection is the interface: each candidate goes in, a score comes out, and the comparison is simply which score is highest. Because the scoring step is separate from the model that generated the candidates, it can judge outputs using criteria that the raw token probabilities do not capture directly.

The source gives several industry illustrations. Stitch Fix and Grab both select the outputs their reward models or verifiers score highly. Nextdoor reported that using a reward model was the key factor in improving the performance of its application in 2023. These examples show that the scoring step is used in production, not only in research.

OpenAI trained verifiers to help models pick the best solutions to math problems. The source reports that a verifier significantly boosted performance, and that the boost was roughly the same as a 30 times increase in model size. In the source's illustration, a 100-million-parameter model with a verifier could perform on par with a 3-billion-parameter model without one.

Selection methods also connect to test time compute, which refers to the computational resources used during inference, when the model is producing outputs rather than being trained. Allocating more compute at inference can mean generating more candidate outputs and then selecting among them. The source notes that scaling test time compute can be more efficient than scaling model parameters, citing DeepMind's research. Every selection method in this article is one way of spending that budget, because each one needs candidates to compare.