Softmax and Converting Logits to Probabilities

A language model does not produce text by looking up a final answer. At each step it produces a score for every token in its vocabulary, and those scores are then turned into probabilities. This article answers the question of what happens next: how the model chooses one token from that probability distribution, and what knob we can turn to change the choice.

The raw scores a model produces are called logits. They are not probabilities yet: they can be any numbers, and they do not add up to 1. Softmax is the step that converts a list of logits into a normalized probability distribution, one value per token, where the values are non-negative and sum to 1. The softmax is applied along the vocabulary dimension, so every token in the vocabulary receives its own probability in the same pass.

The diagram uses made-up logits to show the mechanism; it is not output from the source. The arithmetic behind the probabilities works in two stages. First, each logit is exponentiated, which makes every value positive and enlarges the gaps between the scores. Second, each exponentiated value is divided by the sum of all the exponentiated values, which normalizes them so they add up to 1. Because a larger logit produces a disproportionately larger exponentiated value, the top-scoring token captures most of the probability mass.

Temperature scaling changes the distribution without changing the logits themselves. The model's raw scores stay the same; the scaling happens between those scores and the softmax step. The helper divides every logit by the temperature T, which must be a number greater than 0, and then applies the softmax. Because the division happens first, every logit is rescaled by the same factor before any exponentiation takes place.

The same trace at temperature 1 leaves the logits unchanged, since dividing by 1 does nothing. In that case the probabilities are the ordinary softmax probabilities, and sampling behaves exactly as it would without any temperature scaling. Every other temperature changes the distribution.

Once the probabilities exist, the sampler must choose one token. The deterministic rule always takes the highest-probability token. The probabilistic rule draws a token at random, with a chance proportional to its probability. The difference matters: under the probabilistic rule, a most-likely token is chosen most often but not always.

The source makes this concrete by repeating the random draw 1,000 times and counting how often each token appears. The most likely token, forward, appears in a clear majority of those draws, while several other tokens also appear at least occasionally. A single sampled call can therefore return a different token than the argmax choice, even though the top token still wins most of the time.

The core rule is direct. Temperatures greater than 1 produce more uniformly distributed token probabilities. Temperatures smaller than 1 produce more confident distributions, which are sharper or peakier. A temperature of 1 is the baseline and leaves the distribution unchanged.

The values in the diagram are illustrative, computed from the made-up logits used earlier, not taken from the source. The pattern is what matters: at temperature 5 the top token is still the most likely choice, but the other tokens receive enough weight to be selected much more often.

The effect is not symmetric in what it produces. A very small temperature such as 0.1 makes the draw select the most likely token almost every time, bringing the behavior close to argmax. A large temperature such as 5 adds variety. The source reports that at temperature 5, a continuation such as every effort moves you pizza appeared about 4 percent of the time. Large temperatures add variety at the cost of more nonsensical text.

The scaling happens before the softmax. That ordering determines where to look when a result seems wrong. A mistake in the scaling step shows up as a distorted distribution even when the softmax call itself is correct, so checking only the softmax will not find the problem.

When you trace a program through this pipeline, track three things at each stage: which temperature is active, whether the probability of the most likely token rises or falls relative to the unscaled case, and how many tokens receive any meaningful probability at all. A departure from the expected direction usually points to a step earlier in the list than the one you were examining.