Cross-Entropy Loss for Language Models

When training sequences in a batch have different lengths, the shorter ones are filled out with extra padding tokens so that every row has the same size. The tensors have to be rectangular for the computation to work, but the filler carries no information about the language the model should learn. The problem this article addresses is how to keep that filler from influencing the training signal, specifically the loss that the optimizer uses to adjust the weights.

The approach is to assign a special placeholder value, -100, to every padding position in the target lists. The loss function is then configured to skip any position that carries that value. Padded rows keep their place in the tensor, so the shape never changes, but they contribute nothing to the average. Before explaining the mechanism, it helps to watch the placeholder in action and predict what it does.

Start with a small self-contained example. Each row of the logits tensor holds one score per vocabulary entry for a single predicted position, and the targets tensor holds the index of the correct entry for each row. The example uses two positions and a two-entry vocabulary, which keeps every number easy to check by hand.

Now add a genuine third position with a real target. A correct token ID at that position is supposed to influence the loss. Adding a valid third target changes the loss from 1.1269 to 0.7936. The loss is an average over the positions that count, so a position with a real target shifts the result. The key question is what happens when the third target is a placeholder instead of a real index.

The loss over the three rows equals the loss over the two original rows. Run the code above and compare the two printed values. The third row is still present in the logits, so the tensor shape is unchanged. Only its contribution to the loss is removed. This is the mechanism that lets padded batches be used safely.

PyTorch's cross-entropy loss ignores any target equal to -100. A padded row therefore contributes nothing to the average, even though the tensor shape is unchanged. Each position is checked in turn: if its target is a real index, its loss is computed and counted; if its target is -100, that position is skipped entirely and is not part of the averaged result.

The denominator of the average changes with the ignored positions. The loss is averaged only over the positions that are not ignored. A batch with many padded positions therefore does not dilute the loss with rows that were never meant to count.

Padding in target lists is usually the end-of-text token, ID 50256. Replacing every instance with -100 would look tidy, but it would remove information the model needs. The first end-of-text token in each target list is kept, so the model gets a training target at the stopping point and learns when to generate an end-of-text token in response to an instruction. The model treats that token as the signal that its generated response is complete. Only the later copies, which serve purely as padding, are replaced with -100.

The batching process prepares the target sequence by shifting token IDs one position to the right and appending an end-of-text token. The padding replacement is the last step before those targets reach the loss. The first two stages decide what the target list contains, and the replacement stage decides which positions the loss may see. Only the final stage performs the actual skipping.

This separation matters when you debug a training run. A problem in the padding step will show up later as a loss that looks wrong, even though the loss function itself is working as designed. When a loss looks suspicious, check the target lists before blaming the loss function.