Generating Outputs with AI Models: Sampling Multiple Candidates

A language model rarely has to produce only one answer. When you sample several candidate outputs for the same prompt, a practical question appears: which candidate should the system return? There are two broad options. One is to show several candidates to a person and let that person pick the one that suits them. The other is to build an automatic rule that ranks the candidates and keeps the top one. This article is about the second option, the scoring rules that let software choose.

Every rule covered here starts from the same raw material. Each token in a candidate has a probability that the model computed while generating it, so every candidate carries a trail of numbers. The methods differ in how they combine that trail into a single score, and in one case they replace the trail with a separate scoring model. As you work through the trace below, notice where your own expectation about which candidate wins starts to change.

The most direct selection rule is to pick the output with the highest probability. An output is a sequence of tokens, and the probability of the whole sequence is built from the per-token probabilities. The rule is that the probability of an output equals the product of the probabilities of all the tokens in it. Each new token multiplies the running total by one more factor.

The later factors are conditional. The probability of the second token is written as a probability given the first, and the probability of the third is given the first two. The product therefore reflects the whole chain of choices that produced the sequence, not three independent guesses. This is why the same word can carry a different factor depending on what the model had already written.

Multiplying many small numbers is awkward in practice, so working with probabilities on a log scale is easier. The key fact is that the logarithm of a product equals the sum of the logarithms. Taking the log of a sequence probability therefore turns the chain of multiplications into a chain of additions. The log probability of a sequence is the sum of the log probabilities of its tokens.

Notice what the transformation does to the comparison. Ranking candidates by their probability products and ranking them by their log-probability sums gives the same order, because the logarithm is an increasing function. The sum is simply a more convenient bookkeeping device. Each token adds one more term, and the running total is easy to inspect.

Why are the log values negative? A probability lies between 0 and 1, and the logarithm of any number in that range is negative. Each token therefore contributes a negative amount to the sum, and the more tokens there are, the more negative the total tends to become. This sign fact drives the bias discussed in the next section.

Once outputs are scored by summed log probability, longer outputs face a structural disadvantage. Each additional token adds another negative term, so a long candidate's total usually sits lower than a short candidate's total. If the selection rule simply took the highest sum, it would keep favoring brevity, even when a longer output is the better answer.

The fix is to normalize by length. Dividing the summed log probability by the number of tokens gives the average log probability. This measures how strongly each token contributed on average rather than how many tokens there were. The method then picks the candidate with the highest average. The source reports that this is the approach the OpenAI API uses, as of its writing.

The numbers below are invented for illustration to show the bias. They use the natural logarithm. Suppose a short candidate has two tokens, each with probability 0.5, and a longer candidate has five tokens, each with probability 0.7. Both candidates are plausible, and the question is which one the selection rule should keep.

Probability-based rules depend only on the numbers the model already produced. A different family of methods uses a separate model, called a reward model, to score each candidate. The reward model examines each output and assigns it a score, and the system selects the output with the highest score. The source describes this as the same idea as the verifiers discussed in the previous section.

The flow has four steps. The prompt is sent to the model, several candidates are sampled, the reward model assigns each candidate a score, and the candidate with the highest score is returned. The reward model is what decides the winner, so the candidates' own token probabilities are not needed for the comparison.

The example scores in the diagram are invented for illustration. The source does not give scores for specific candidates. Its point is the structure: several candidates go in, one score per candidate comes out, and the maximum is kept.

The source gives several industry illustrations. Stitch Fix and Grab both select the outputs their reward models or verifiers score highly. Nextdoor reported that using a reward model was the key factor in improving the performance of its application in 2023. These examples show that the scoring step is used in production, not only in research.

OpenAI trained verifiers to help models pick the best solutions to math problems. The source reports that a verifier significantly boosted performance, and that the boost was roughly the same as a 30 times increase in model size. In the source's illustration, a 100-million-parameter model with a verifier could perform on par with a 3-billion-parameter model without one.

These selection methods connect to a broader idea called test time compute, which refers to the computational resources used during inference, when the model is producing outputs rather than being trained. Allocating more compute during inference can mean generating more candidate outputs and then selecting among them. The source notes that scaling test time compute can be more efficient than scaling model parameters, citing DeepMind's research.

The source frames this with a question: if a language model is allowed a fixed but nontrivial amount of inference-time compute, how much can it improve its performance on a challenging prompt? Every selection method in this article is one way of spending that budget, because each one needs candidates to compare.