Hallucination in Language Models

Ask a language model the same question twice and you may receive two different answers. Sometimes that variety is useful. A brainstorming tool may want surprising ideas. Other times it is a problem. A customer-facing product needs answers that are consistent and correct. Incorrect or misleading output of this kind is called hallucination, and it is one of the main reasons that controlling how a model's output is produced matters.

Sampling methods are the practical choices that control how a model's output is drawn from its probability estimates, and how several outputs can be produced and compared. This article works through the three main ones: temperature, log probabilities, and self-consistency. Keep in mind that each tool answers a different question. Temperature controls the randomness of each individual choice. Log probabilities are a window into the model's confidence. Self-consistency uses several draws and a scoring rule to pick one result.

A useful habit while reading this article is to ask, at each step, where the randomness enters and what control you have over it.

A language model produces output one token at a time. At each position it estimates how likely each possible next token is. Temperature is the setting that controls how random the model's choice from those estimates is. The summary of the material is direct: temperature controls how random a model's output is, and higher values let lower-probability tokens surface more often.

Read that carefully. Temperature does not make the model choose randomly among all tokens equally. It shifts how much the choice depends on the estimated likelihoods. When temperature goes up, the high-likelihood tokens still tend to be chosen, but the low-likelihood tokens appear more often than they otherwise would. That is the mechanism behind more varied and sometimes more surprising output.

The diagram shows the qualitative change only. The source describes this as an intuition about chaos rather than an exact measurement, so do not expect a fixed numeric formula for how much a given temperature moves each token. What you can rely on is the direction of the change: higher temperature gives lower-probability tokens more chances to appear.

Why use a logarithm at all? Probabilities for individual tokens can be extremely small. A small number has to be stored in some numeric format, and that brings a practical problem. Working with logarithms keeps these values in a form that is easier to compare and combine. A consequence worth remembering is that a probability between 0 and 1 has a logarithm that is negative. The closer a probability is to 1, the closer its log is to 0. A very unlikely token has a log probability that is a large negative number.

This is why a log probability works as a window into the model's confidence. Reading the value tells you how strongly the model favored a token. Values near zero correspond to tokens the model considered likely, and more negative values correspond to tokens it considered unlikely.

The limit is about what is returned, not about what the model computes. The API returns a slice of the full distribution, namely the most likely tokens at each position, while the model's internal distribution covers every possible token. Anything outside that returned slice is invisible to you.

A single token's log probability tells you about one choice. To judge a whole candidate output, you need one number for the entire output. The approach described in the material is to average the logprobs of the output's tokens. Averaging gives a per-token score, so outputs of different lengths can be compared on a comparable scale.

In this formula, $n$ is the number of tokens in the candidate output and $\log p_i$ is the log probability of the $i$-th token. A higher average, meaning a value closer to zero, indicates that the model was more confident across the output as a whole.

Self-consistency is a technique for improving the quality of generated output. Instead of accepting the first output, you generate several outputs for the same input and select the best one. The approach was named by Wang et al. (2023). The best candidate is chosen by a scoring rule, and the rule connected to this technique is the average logprob of each output.

Trace the process in three phases. First, sample several outputs for one input. Second, compute each output's average logprob using the averaging described above. Third, compare the averages and keep the output with the highest one. The result is a single output that the model's own confidence favored among the candidates.

In the OpenAI API, the parameter best_of carries out this process for you. The material's example sets it to 10, which asks the model to return the output with the highest average logprob out of 10 different outputs. You do not write the comparison yourself in that case, but knowing the three phases above tells you what the parameter is doing.

Generating several outputs costs more than generating one, because paid model APIs often charge per output token. Some of that cost can be reduced, for instance by processing the input once and reusing it across all of the outputs instead of processing it again for each one.