Logits and Softmax Transformation

When a language model is asked what should come next, its first output is a list of numbers. A common first reaction is to read each number as the chance that a particular token comes next. The numbers look like scores that rank the options, and a larger number seems to mean a more likely token. Part of that intuition is correct. The belief that the raw numbers already are probabilities, however, is a misconception. The raw numbers are called logits, and they are a different kind of quantity.

For each input the model processes, it produces a logit vector. This is simply an ordered list of numbers. Every position in that list corresponds to one token in the model's vocabulary. If the vocabulary has N tokens, the logit vector has N entries, and each entry is the model's raw score for the token at that same position. The vector is therefore a complete snapshot of the model's current leaning across the whole vocabulary, taken at one moment for one input.

Because the position of each number is tied to a specific token, the order of the list matters. Reading logit number 2 means reading the score for vocabulary token 2, not some generic value. Keeping that correspondence in mind makes the vector easy to interpret, because each number answers the question of how strongly the model leans toward one particular token. Shuffling the numbers without shuffling the tokens would attach every score to the wrong word.

Logits share one feature with probabilities. A larger logit corresponds to a higher probability once the scores are converted. The resemblance ends there, because logits differ from probabilities in two specific ways. First, logits can be negative, while probabilities must be non-negative. Second, logits do not sum to one, while the probabilities across the vocabulary must total one. These two differences explain why the raw output must be transformed before it can be read as a distribution of likelihoods. The job of that transformation is performed by a softmax layer.

To convert logits to probabilities, a softmax layer is often used. Suppose the model has a vocabulary of N tokens and the logit vector is the list of values x_1 through x_N. The probability for token i, written as p_i, is computed from the logit at position i. The softmax step has two parts. It first exponentiates every logit, and then it divides each exponentiated value by the sum of all the exponentiated values.

Each part of the formula solves one of the two problems from the previous section. The exponential step turns every logit into a positive number, so a negative logit no longer produces a negative result. Dividing each result by the total of all of them forces the outputs to sum to one. Both properties are exactly what probabilities require. The formula reads the same way for every token, so the same two operations apply to the whole vocabulary.

Read the diagram as a change in the values, not in the tokens. Each token keeps its position and its identity. The negative score of -1.0 becomes a small positive probability, and the three values now total one. The ranking survives the transformation, because 2.0 was the largest logit and 0.705 is the largest probability.

The following errors come up often when learners first meet raw model output. Each one follows from treating logits as if they were already probabilities.

Across the example, the logit vector kept its meaning at every stage. Each position stayed tied to one vocabulary token, the raw values were allowed to be negative and to total any amount, and softmax then converted them into a valid distribution. Only after that final conversion can the numbers be read as the model's probabilities for each token.