Batching Instruction Examples with a Custom Collate Function

By the time you reach the end of the instruction fine-tuning chapter, you already have most of the pieces in place. You have built an InstructionDataset class that holds instruction examples, you have written a custom_collate_fn function that batches those examples, and you have fine-tuned a model. The final exercise, Exercise 7.2, asks for one more move: change which tokens count during training, and then judge whether that change actually helps.

Think of this as a small experiment layered on top of a pipeline you already understand. Nothing new is being built from scratch. A single variable, the masking of instruction and input tokens, is being altered, and the result is compared against an earlier run. That framing matters, because it turns a vague question (is this change good?) into a concrete one (did the score move, and was the movement caused by masking?).

Start with a single tokenized instruction example and follow it through the masking step. Every position in the example has a label, and masking decides which labels keep their real token IDs and which are replaced. The diagram below shows one example with six token positions. The first three positions hold the instruction and input, and the last three hold the response the model should learn to produce.

The value -100 works as a marker rather than a token. Positions carrying -100 are positions the training setup treats as not counting. That is why masking is a choice about which tokens count during training: the same example is kept intact, but the marked positions are excluded from what the model is asked to reproduce. Exercise 7.2 then asks whether this selective treatment has a positive effect on model performance.

Masking only matters if the masked examples actually reach the model in batches, so the next question is how data travels from storage to training. The source describes three roles. The InstructionDataset object holds the instruction examples. The custom_collate_fn function describes how individual examples are combined into a batch. The PyTorch data loader connects the two and handles shuffling and batching, so the training code can simply request the next batch.

In practical terms, the InstructionDataset object is plugged into a PyTorch data loader together with the custom_collate_fn function. The loader asks the dataset for examples, shuffles them, groups them into batches, and passes each group through the collate function. The training loop then receives a ready batch and never has to handle individual examples itself.

The collate function is where unequal examples become one rectangular batch. The diagram shows the generic idea: examples of different lengths are brought to a shared length and stacked. Padding positions in the labels are conventionally given the -100 value, so they are excluded in the same way that masked instruction tokens are. The source does not spell out this padding convention in detail, so treat the padding placement as the general pattern to verify against your own collate function.

The source presents instruction fine-tuning as three stages: preparing the dataset, fine-tuning the LLM, and evaluating the LLM. Each stage contains numbered steps. Creating the data loaders is the closing move of the first stage. It is the point where prepared data stops being a collection of examples and becomes a stream of batches that the fine-tuning stage can consume.

Data loaders are created for the training, validation, and test sets. Each split gets its own loader, so the same batching logic serves both fine-tuning and evaluation. Because of this handoff, everything in stage 2 depends on the loaders built at the end of stage 1. The masking experiment depends on the same handoff, since the data that flows through those loaders is what the masked training run actually sees.