Temperature and Output Randomness

Ask a model the same question twice and you may get two different answers. Sometimes that variety is useful, and sometimes it is a problem. A customer-facing product needs answers that are consistent and correct, while a brainstorming tool may want surprising ideas. The difference between those two needs is largely a matter of how a model's output is drawn from its probability estimates, and that is the subject of this article.

Before reading on, make a prediction. Suppose you raise the temperature setting. Think about which tokens, the high-probability ones or the low-probability ones, would be more likely to show up in the output.

A model produces output by making a series of choices, one token after another. At each step it has a set of candidate tokens, each with an estimated probability. Sampling methods are the practical choices that control how the next token is drawn from those estimates, and how several complete outputs can be produced and compared.

The source frames the whole topic around one question asked at each step: where does the randomness enter, and what control do I have over it? It names three tools, and each answers a different question. Temperature controls the randomness of each individual choice. Log probabilities are a window into the model's confidence. Self-consistency uses several draws and a scoring rule to pick one result. Keeping these three apart in your mind makes the rest of the article easier to follow.

Temperature controls how random a model's output is. Higher values let lower-probability tokens surface more often. Lower values keep the output closer to the tokens the model rates most likely. The effect is applied to each choice, so a higher temperature gives many small chances for an unlikely token to win, and those chances add up over a long output.

Read the diagram as a before and after of the same two tokens. The likely token still exists after the change, but it no longer dominates as strongly. The unlikely token gains share, which is exactly what the source means when it says higher temperature lets lower-probability tokens surface more often. The diagram is an illustration of direction, not a measurement of specific probabilities.

Why use the logarithm at all? The source does not spend much time on that, but the practical point is that the values line up with confidence in a readable way. A token the model considers very likely has a probability close to 1, and its log probability is close to 0. A token the model considers very unlikely has a probability close to 0, and its log probability is a large negative number. So a log probability near zero signals a likely token, and a more negative value signals a less likely one.

Keep this mapping in mind when you read logprob values. You are not looking at a score out of 100 where bigger is better in every sense. You are looking at a logarithm of a probability, so values closer to zero mean the model rated that token as more likely.

Not every provider makes logprobs available, and the ones that do may limit what you can see. As of the source's writing, the OpenAI API showed logprobs only for up to the 20 most likely tokens at each position. It had once allowed retrieving logprobs for arbitrary text supplied by the user, but that option was discontinued in September 2023. Anthropic did not expose logprobs for its models, according to the source.

Probabilities for individual tokens can be extremely small, and a small number has to be stored in some numeric format. The underflow problem occurs when a number is too small to be represented in a given format, so it gets rounded down to zero.

The danger is that once a value has been rounded to zero, the information that it was small but not impossible is gone. Anything that depends on comparing those tiny values can no longer tell them apart. Two tokens that were very different in how unlikely they were may end up looking identical, which is why the source treats the boundary between tiny and impossible as something worth preserving.

Self-consistency is a technique for improving the quality of generated output. Instead of accepting the first output, you generate several outputs for the same input and select the best one. The source notes that Wang et al. (2023) called this approach self-consistency. The best one is chosen by a scoring rule, and the rule the source connects to this technique is the average logprob of each output.

In the OpenAI API, the best_of parameter can be set to a specific value to carry out this process. The source's example sets it to 10, which asks the model to return the output with the highest average logprob out of 10 different outputs. You do not write the comparison yourself in that case, but it helps to understand what the parameter is doing.

The comparison score in self-consistency is the average of the per-token log probabilities across a whole output. Averaging puts outputs of different lengths on a common footing: you divide the total by the number of tokens, so a long output is not penalized only for having more tokens. Each per-token value is a log probability, so values closer to zero indicate more likely tokens, and a higher average means a more confident output overall.

Generating several outputs costs more than generating one, since paid model APIs often charge per output token. Generating ten candidates means paying for roughly ten outputs' worth of tokens, so the technique is a deliberate trade of cost for a chance at a better result.

The source points out that some of that cost can be reduced. For instance, the input might be processed once and reused across all of the outputs, rather than being processed again for each one. This is a useful thing to ask about when you evaluate a provider or a method.

Sampling controls only go so far. The source describes a model whose structured output is unreliable, and reports that the percentage of correctly generated JSON objects, depending on the application and model, ranged anywhere from 0% up to the high 90s. Turning temperature or scoring rules does not repair a model that cannot produce the format at all.

When a model is this brittle, the source's suggested fix is to replace it with a different model rather than to keep tuning the sampling controls. Another route the source mentions is training a model from scratch on data in the desired format, though the source says that is outside its scope. Fine-tuning, which adjusts a pre-trained model to better suit a specific task or dataset, is related. Some fine-tuning services adjust the model to the desired format automatically.

Hallucination, meaning the generation of incorrect or misleading information, is one reason these controls matter. In a review of three months of customer support requests for an AI company, the source's author found that one-fifth of the questions concerned handling the inconsistency of AI models. In a 2023 panel, the speakers agreed that hallucination is the biggest blocker for many AI enterprise use cases.