Organizing Data into Training Batches

Once an instruction dataset has been downloaded and formatted, it is still one undivided list of examples. Before any batches can be built for instruction fine-tuning, that list has to be divided into separate pieces. This article is about that division and about the step that follows it, grouping the entries of each piece into batches. The source material treats the work as a familiar one: it mirrors the approach used with the spam classification dataset in the previous chapter, so the logic should feel familiar even though the data is now instruction-style text.

The phrase organizing instruction data in training batches has two parts, and it helps to separate them before thinking about any mechanism. The first part, instruction data, is the material being organized. It is the set of instruction records, each one an example the model will learn from. The second part, training batches, is the grouping that the organizing is done into. In other words, the records are the unit being moved, and the batch is the container they are moved into.

The relationship between the two is containment. A collection of instruction data is divided into groups, and each group is a training batch. Nothing is lost or duplicated in the process: every record ends up in exactly one batch. The batch is therefore the unit to keep in view when you reason about the data, because the whole collection is organized by being split into such groupings.

The source organizes the whole fine-tuning workflow into three stages. Stage 1 prepares the dataset, Stage 2 fine-tunes the model, and Stage 3 evaluates it. Partitioning belongs to the preparation work in Stage 1. It comes after the dataset has been downloaded and formatted, and it comes before the data loaders are built.

The order matters because of dependency. The data loaders are built from the prepared portions, so the portions must exist first. Batching then groups entries from those portions into the units the model processes during training. Put simply, the dataset is divided into portions first, and only then is one portion divided into batches. These are two different divisions, and confusing them is one of the most common misreadings covered later in this article.

The partitioning starts from one number: the total length N of the data list. The training and test sizes are computed as proportions of that length, and each product is wrapped in int(), which keeps only the whole-number part. The validation size is not a proportion at all. It is whatever remains after the training and test sizes are subtracted from the total.

Once the three sizes are known, each slice follows from them. The training slice starts at the beginning of the list and stops before train_portion. The test slice starts at train_portion and runs for test_portion entries, so it ends just before train_portion plus test_portion. The validation slice starts at that combined offset and continues to the end of the list. Read in this order, the three slices cover the list from its first entry to its last, with no overlap and no gaps. The test boundary is the start of the test slice, and the validation boundary is the start of the validation slice; each one is the end of the slice before it.

The code above is a reconstruction of the logic the source describes: it computes three integer sizes from the total length, slices the list into three pieces, and prints the length of each piece. The source's own listing assumes that a list called data already exists and holds every instruction example. Read the code once for its overall shape, then trace it line by line, watching the moment each variable receives its value and the moment each slice is cut.

Read the diagram from left to right. Each portion begins exactly where the previous one stops, so the boundaries are the only places where one slice hands over to the next. The training portion is the largest, the test portion comes next, and the validation portion occupies the final positions in the list. This is a statement about position in the list. The name of a portion tells you what it is for, but its slice tells you exactly which entries it contains.

The same rules apply to any length. Try the procedure yourself on a much smaller list before reading the answer. Suppose the data holds six instruction records, labeled A through F. Work out how many entries each portion receives, and which records fall where.

After a portion exists, its entries can be grouped into batches. The grouping is the same containment idea described earlier: the records are placed into groups, and each record belongs to exactly one group. The example below is a generated illustration of the grouping idea, not a procedure from the source. It uses a small set of six instruction records labeled A through F, organized into batches of two.

The diagram shows only the containment relationship the source names. Records sit inside batches, and batches sit inside the collection. In the full workflow, the collection being batched is one of the portions from the earlier split, not the whole original dataset. The batch sizes shown here are an illustration; the source does not provide a batch size for the course's own dataset.

The partition arithmetic is short, but a small slip in it silently changes which entries land in which portion. The comparison below shows what happens when the test slice is given the wrong end. The correct version stops the test slice at index 1045, which leaves exactly the validation entries for the validation slice. The wrong version lets the test slice run to the end of the list.