Training a Model on Instruction Data with Loaders

Each split is loaded by creating an InstructionDataset for it and then wrapping that Dataset in a DataLoader.

Programming

Three Splits, One Repeated Pattern

Instruction fine-tuning works with three separate splits of data: training data, validation data, and test data. Each split has to be fed to the model in batches, and the code that does this follows one repeated pattern. For each split, you first wrap the raw data in a custom Dataset object, then pass that Dataset to PyTorch's DataLoader along with the settings that control how batches are formed.

The goal of this article is not to memorize a setup listing line by line. The goal is to learn to read such a listing so that, for any loader, you can tell which data it reads, how it groups that data into batches, and how its ordering and parallelism are set. Reading a loader is a skill you can apply to any of the three splits.

Reading the Loader Call Before Anything Runs

Before reading further, look at the shuffle argument in each DataLoader call and make a prediction. Think about what shuffle=True would do to the order in which a loader hands out examples, and whether that order matters for each split. Write down your guess before you read the explanation that follows.

What do you think happens?

A dataset has 7 samples and batch_size is 3, with drop_last=False. How many batches does one full pass through the loader produce?

Reveal answer

Answer:

Seven samples split into groups of three gives two full batches of three and one final batch holding the single leftover sample. With drop_last=False that leftover batch is kept, so there are 3 batches in total.

Dataset and DataLoader Roles

Definition

A DataLoader is a PyTorch component that wraps a Dataset and produces batches from it. It requires two inputs: a Dataset instance and a batch_size. The Dataset tells the loader where the examples come from, and batch_size tells it how many examples to group together.

In this setup, the Dataset for each split is an InstructionDataset, a custom Dataset class built for instruction data. Its constructor takes the raw split and a tokenizer, as in InstructionDataset(train_data, tokenizer). The DataLoader does not itself know how instruction data is stored. It relies on the Dataset to supply individual examples, and it adds the grouping, ordering, and parallel-loading behavior on top.

What the DataLoader wraps, what the Dataset provides to it, and what the DataLoader adds on top

Following One Split Through a Loader

The following illustrative code shows the two-step pattern for one split. The names and values are chosen for teaching and are not taken from a specific source listing. The first line creates the Dataset from the training split, and the second wraps it in a DataLoader with the five keyword arguments that follow the dataset.

python

train_dataset = InstructionDataset(train_data, tokenizer)
train_loader = DataLoader(
    train_dataset,
    batch_size=batch_size,
    collate_fn=collate_fn,
    shuffle=True,
    drop_last=True,
    num_workers=num_workers
)

train_dataset = InstructionDataset(train_data, tokenizer)
train_loader = DataLoader(
    train_dataset,
    batch_size=batch_size,
    collate_fn=collate_fn,
    shuffle=True,
    drop_last=True,
    num_workers=num_workers
)

Notes on the code

train_dataset: the Dataset that supplies individual training examples
batch_size: how many examples are grouped into each batch
collate_fn: assembles the individual samples into one batch
shuffle=True: changes the order in which examples are handed out
drop_last=True: controls whether an incomplete final batch is kept
num_workers: sets how many worker processes prepare data

Notice that the call has exactly one positional input, the Dataset, and the rest are keyword arguments. The Dataset and batch_size are the two inputs every loader in this setup requires. The remaining keywords adjust how batches are formed and loaded.

A generated illustration with 7 samples and batch_size 3. Each sample index is shown with the batch it falls into when drop_last=False.

The Five Keyword Arguments

Every DataLoader call in the setup passes the same five keyword arguments after the dataset. Each one changes a different part of the loading behavior. The table lists them with the role each one plays.

Argument	What it controls
batch_size	How many examples are grouped together; required along with the Dataset
collate_fn	How individual samples are combined into one batch
shuffle	Whether the order in which examples are handed out is changed
drop_last	Whether an incomplete final batch is kept or discarded
num_workers	How many worker processes load data in parallel; 0 means no parallel processing

The five keyword arguments passed after the dataset in each loader call

What Each Loading Setting Changes

Three of the five arguments change behavior in ways that are easy to misread: shuffle, drop_last, and num_workers. The three subsections below take them one at a time.

Mistakes Beginners Make with Loader Settings

Common mistake
Shuffling the validation or test loader
What it looks like
```
val_loader = DataLoader(val_dataset, batch_size=batch_size, collate_fn=collate_fn, shuffle=True)
```
Why it fails
The validation and test loaders in the setup use shuffle=False. Changing the order of evaluation data is not what those splits are configured to do.
Do this instead
Keep shuffle=False for validation and test loaders, as the setup does.
Common mistake
Assuming drop_last=True only changes the speed of loading
Why it fails
drop_last changes which samples reach the model. An incomplete final batch is discarded rather than processed.
Do this instead
Decide drop_last based on whether every example must be included in that pass.
Common mistake
Expecting num_workers to change batch contents
Why it fails
num_workers controls parallel data loading. The grouping of examples into batches is set by batch_size, shuffle, and drop_last.
Do this instead
Read batch contents from batch_size, shuffle, and drop_last, and read loading parallelism from num_workers.
Common mistake
Passing a raw list of examples where a Dataset instance is expected

Comparing the Three Loaders

The three loaders share a batch size and a collate function. The differences come down to the dataset each one wraps and the two flags that change ordering and batch completeness. The comparison below lists the settings side by side.

Batch size and collate function are the same for every loader; shuffle and drop_last differ between the training loader and the validation and test loaders

Worked example

Choosing settings for a new validation loader

You need a validation loader for a validation split. It must share batch_size and collate_fn with the training loader. Which of the five arguments should change?

1
Identify the shared settings
batch_size and collate_fn are the same as in the training loader, so they are passed unchanged.
2
Set ordering
Validation data is configured with shuffle=False in the setup, so shuffle is set to False.
3
Set batch completeness
The validation loader uses drop_last=False, so no validation examples are discarded.
4
Set parallelism
num_workers is passed the same way as in the other loaders, since it controls loading and not batch contents.

Result

The validation loader uses the same batch_size, collate_fn, and num_workers as the training loader, with shuffle=False and drop_last=False.

Practice Reading Loader Settings

MEDIUM

A loader is created with batch_size=4, shuffle=False, drop_last=True, and num_workers=0 on a dataset of 10 samples. Which samples are handed out in the final batch, and is that batch handed out at all? Explain using drop_last and the sample count.

Hints

Divide 10 samples into groups of 4 and see what is left over.
Check the drop_last value to decide whether the leftover group is kept.

Your attempt (not saved)

Key Takeaways

Each split is loaded by creating an InstructionDataset for it and wrapping that Dataset in a DataLoader.
A DataLoader requires a Dataset instance and a batch_size; collate_fn, shuffle, drop_last, and num_workers adjust how batches are formed and loaded.
The training loader uses shuffle=True and drop_last=True, while the validation and test loaders use shuffle=False and drop_last=False.
num_workers controls parallel data loading, where 0 means no parallel processing and higher values depend on operating system support.
The three loaders share batch size and collate function, so reading a loader means identifying its dataset and its ordering, completeness, and parallelism settings.

Related concepts

Instruction Fine-Tuning Workflow

More concepts starting with T