Cross-Entropy Loss in Language Model Training

Imagine you are preparing a dataset to fine-tune a large language model so that it follows instructions. Each training example has three parts: an instruction that describes a task, an optional input the task operates on, and the response you want the model to produce. Once the text is assembled, the model is asked to predict the tokens that come next, one position at a time, and it is adjusted whenever its predictions drift from the expected tokens.

That setup raises an uncomfortable question. The instruction is part of the text the model sees, so it is also part of what the model is asked to predict. If nothing special is done, a share of the training effort goes toward reproducing the wording of instructions, which is not the skill you actually want the model to acquire. Instruction masking is a technique for steering that effort away from the prompt and toward the answer.

To see the mechanism, follow one short example through the target sequence. Suppose the formatted example contains an instruction that reads Translate to French, an input word Hello, and a response Bonjour. The numbers below are illustrative placeholders, not real vocabulary IDs. Before masking, every position in the target holds its real token ID. After instruction masking, the instruction positions are overwritten with -100, while the response positions keep their IDs.

Read the diagram as a rule about the target alone. The three instruction positions and the input word are overwritten, so they contribute nothing to the loss. Only the response token Bonjour keeps its ID and is graded. The illustrative ID 2214 stands in for whatever real ID the tokenizer assigns to that word. The rule is not about deleting text: the same tokens still exist in the input that the model reads.

Preparing a training example involves two parallel versions of the same formatted text. The first is the input text, which is tokenized and fed to the LLM during training. The second is the target text, which is what the model's predictions are compared against. Before masking, both versions are essentially identical, and the target is simply the input shifted so that each position is paired with the token that should follow it.

The two versions share positions. Position by position, the input and the target match until masking begins to overwrite the target. The model's predictions come from the input, while the comparison is made against the target. Because the instruction positions are overwritten in the target, the loss has no expected value to compare against at those positions, so they do not count toward learning. The model still reads them as context for the positions that follow.

When a position in the target is masked, its token ID is replaced by the value -100, which is the ignore_index value. The name describes its role: a loss calculation configured to ignore that index skips the position entirely. Because the replacement happens in the target and not in the input, the model still reads the masked tokens as context for the positions that follow them.

Masking is closely related to a more common step: masking padding tokens. Padding is added so that training examples in a batch share the same length, and those padding positions are usually masked for the same reason. Instruction masking extends that idea to a meaningful part of the content itself. It is therefore a choice about which positions the loss is allowed to see, not a change to the text the model reads.

Compare the two training setups on the same example. Without masking, every target position has a real ID, so the loss pushes the model to predict the instruction wording as well as the response. With masking, only the response positions carry a real ID, so the loss is shaped only by how well the model predicts the answer. The main motivation is a shift in what the model is rewarded for: with masking on, the model is trained to focus on producing accurate responses rather than memorizing the instructions it is given.

The source identifies reduced overfitting as the benefit masking can offer. Overfitting is the general problem of a model fitting the training data too closely at the expense of generalizing. Masking does not guarantee that outcome; it changes the signal, and the effect has to be judged by how the trained model performs.

Masking is a reasonable default to reason about, but it is not settled. The source notes that researchers are divided on whether masking instructions is universally beneficial during instruction fine-tuning. It cites a 2024 paper by Shi et al., titled Instruction Tuning With Loss Over Instructions, which demonstrated that not masking the instructions can benefit LLM performance.

The source's own example leaves masking off and treats turning it on as an optional exercise for readers. That choice is a useful reminder that the technique is a tool you can enable, test, and compare, rather than a requirement built into every fine-tuning run.