Batch Size Trade-offs in Training

So far every batch drawn from the data loader has contained a single sample. That keeps the picture simple, but it hides how the loader behaves when its settings change. The practical way to build intuition is to run the same loader several times with different settings and watch what changes in the output. This article uses that approach. You will trace how one call is executed, predict the shape of a batch before running it, and learn where to look when a printed batch does not match what you expected.

The source uses a loader with batch_size=8, max_length=4, and stride=4, then pulls out exactly one batch. The code below follows that pattern. Shuffling is turned off, so the printed rows follow the order the loader produces them.

Read the code in the order Python runs it. The first statement only configures the loader. The second statement creates an iterator, which is a handle that will produce batches on request. The third statement is the one that actually draws a batch, so it is the step where batch_size takes effect. The two print statements come last, so they show whatever that single batch contains.

Before running the batch_size=8 example, answer a structural question using only the numbers in the call. The contents of raw_text are not needed, because the shape of the batch follows directly from the settings.

The source suggests two pairs of settings for building intuition: max_length=2 with stride=2, and max_length=8 with stride=2. Both pairs keep stride fixed at 2 while max_length changes. Holding one value constant and varying the other is a useful habit, because any difference in the output can then be traced to the single setting you changed.

Treat the two settings as two separate questions. max_length asks how long each sample is. stride asks how far the loader moves forward before it starts the next sample. The two settings are separate knobs, but they cooperate: max_length determines how much text one sample contains, and stride determines where the next sample begins.

When the two values are equal, as in the example with max_length=4 and stride=4, each sample starts exactly where the previous one ended. When they differ, the starting positions shift, and the samples can overlap or leave gaps. The diagram below shows positions 0 through 7 for the equal case, with each sample grouped.

The batches sampled earlier had a batch size of 1. The source notes that this is useful for illustration. With one sample per batch, you can see exactly what the loader produces without any grouping to interpret. Once batch_size rises above 1, the loader groups several samples together in each call to next, and the output becomes a block of rows instead of a single row.

The source describes the trade-off that governs the choice of batch size. Small batch sizes require less memory during training, but they lead to more iterations. Larger batches reverse that balance. The comparison below uses only those points, plus the direct reading of the batch_size=8 example.

When a printed batch does not match what you expected, the useful question is where in the chain the two parted ways. Work through the chain in order. Most divergences can be located in one of four places, and the order below is the order to check them.