Creating a Custom Dataset for Instruction Data

Instruction fine-tuning works with three separate splits of data: training data, validation data, and test data. Each split has to reach the model in batches, and the code that does this follows one repeated pattern. For each split, you first wrap the raw data in a custom Dataset object, then pass that Dataset to PyTorch's DataLoader along with the settings that control how batches are formed.

The goal of this article is not to memorize a setup listing line by line. The goal is to learn to read such a listing so that, for any loader, you can tell which data it reads, how it groups that data into batches, and how its ordering and parallelism are set.

Each split follows the same two steps. First, a dataset is created from the split. In the listing this article is based on, that Dataset is an InstructionDataset, a custom Dataset class built for instruction data. Its constructor takes the raw split and a tokenizer, as in InstructionDataset(train_data, tokenizer). Second, that dataset is wrapped in a DataLoader, which produces the batches the model consumes.

The DataLoader does not know how instruction data is stored. It relies on the Dataset to supply individual examples. The Dataset answers the question where do the examples come from, while the DataLoader answers how many examples go together and in what order they are handed out.

A DataLoader needs two inputs: a Dataset instance and a batch_size. The Dataset tells the loader where the examples come from, and batch_size tells it how many examples to group together. The other keyword arguments adjust how batches are formed and loaded. In the listing this article follows, every DataLoader call passes the same set of keyword arguments after the dataset, and each one changes a different part of the loading behavior.

To read a call, start with the first argument to find the dataset, then check batch_size for the group size. Next, look at shuffle and drop_last, which decide the order and the completeness of the batches. Finally, look at num_workers to see whether loading happens in parallel. Reading the call in this order lets you answer for any loader which data it reads and how that data is grouped.

Consider a training split with 7 examples, numbered by their dataset index from 0 to 6, and a loader with batch_size=3, shuffle=True, and drop_last=True. Shuffling changes the order in which indices are visited, so the same examples may be grouped differently on a different pass. The grouping below is one possible result for a single pass.

The diagram shows the key point of this section. Shuffling decides which indices share a batch, and drop_last decides what happens to the remainder. Index 3 is not lost because of the shuffle itself; it is lost because the number of examples did not divide evenly into full batches and drop_last discarded the incomplete group.

num_workers controls parallel data loading. A value of 0 means there is no parallel processing: examples are loaded in the main process. Higher values allow loading to be spread across worker processes, and whether those higher values work depends on operating system support.

The path a batch takes is the same regardless of num_workers. Examples come from the Dataset, pass through the collate function that assembles them into one batch, and the resulting batch is handed to the training loop. What num_workers changes is where the reading of examples happens: with 0 it happens in one process, and with more workers the reading can proceed in parallel before the batches reach the training loop.

The diagram is a simplified model. The source confirms only that num_workers controls parallel loading and that 0 means no parallel processing. It does not describe the internal mechanics of worker processes, so treat the middle node as the place where num_workers acts, not as a full description of how it is implemented.

The three loaders share a batch size and a collate function, so the differences between them come down to the dataset each one wraps and the two flags that change ordering and batch completeness. The training loader shuffles and drops incomplete batches, while the validation and test loaders keep a fixed order and keep every example.

The comparison shows that only the training column differs in its loading flags. A reader who understands this pattern can look at any new loader call and check first whether it matches the training pattern or the evaluation pattern.