Batch Size and Training Stability

Before looking at any code, start with the question the code answers: how many training examples should the model see before it adjusts its weights? In regular deep learning, that number is the batch size. This article answers that question first, then shows how a data loader turns a long stream of text into batches whose overlap you can predict and control.

Two ideas run through the whole article. The first is that batch size is a tradeoff, not a fixed rule. The second is that the way windows are placed inside the text, controlled by the stride, determines whether one batch repeats tokens from the last one or leaves tokens out.

Training on very small groups produces updates that jump around, because each group is a rough sample of the whole dataset. A small batch sees only a few examples, so the direction it suggests for changing the weights can differ quite a lot from the direction a full view of the data would suggest. Those jumps are the noise.

Training on very large groups produces smoother updates. The price is that each update costs more, and the model sees fewer updates per pass over the data. Batch size therefore sits in the middle of a tradeoff: smoothness and cost pull in opposite directions.

The loader is built by a function called create_dataloader_v1. It takes five arguments: the raw text, the batch size, the maximum length of each sequence, the stride, and a shuffle flag. Each argument controls a different part of the batch.

Notice that the call asks for a batch size greater than 1. Each call to next pulls one whole batch, so the inputs and targets come back as grids in which each row is one sequence. Keep that row structure in mind, because the question of overlap is really a question about which rows share tokens.

Before you look at what the loader returns, make a prediction. Consider a call where each sequence is four tokens long and the stride is also four. Think about where the second sequence begins relative to the first.

If your guess was that sequences repeat, you were picturing a window that moves more slowly than its own length. That mismatch is exactly what the stride controls, and the next section makes it concrete.

The stride is the distance the loader moves forward between one sequence's starting token and the next sequence's starting token. The window length is set by the maximum length, and the stride is set separately. Because these two numbers are independent, the windows can sit end to end, overlap, or leave gaps.

When the stride is smaller than the maximum length, each window reuses some tokens from the window before it. When the stride equals the maximum length, the windows sit end to end. The source states the result directly: increasing the stride to 4 uses the data set fully and avoids any overlap between batches. It also notes that more overlap could lead to increased overfitting, so overlap should be chosen deliberately.

The loader returns two grids, inputs and targets, and both are built from the same windows. When you compare a row of inputs with the matching row of targets, the targets are the same tokens shifted forward by one position. That is what lets the model learn to predict the next token from the tokens that came before it.

Batch size changes how many such row pairs come back together, but it does not change the pairing inside each row. Each row is always its input window matched with its target window shifted by one.

Suppose a batch you print contains repeated tokens from one sequence to the next, but you expected none. The useful debugging question is where the divergence starts. Start with the arguments, not the model. The flowchart below shows the order of checks.