Input Windows and Batch Creation

When a language model is trained on a long piece of text, the whole text cannot be fed in all at once. It has to be divided into smaller pieces called batches, and each batch is built from a portion of the input dataset. The way those pieces are chosen determines which words the model sees together and how much of the text repeats from one piece to the next. The idea that controls this behavior is called stride.

Stride is the number of positions by which the input window shifts when the next batch is created from an input dataset. The rest of this article follows a single question: when the window slides forward, how far does it move each time, and what does that do to the batches it produces?

Imagine placing a small frame over a passage of text. You read only what falls inside the frame, then slide the frame forward and read the next portion. The frame is the input window. Whatever is visible inside it at one moment is one batch. Each time you slide the frame, you produce a new batch.

Two separate settings control this process. The window size sets how many items a batch contains. The stride sets how far the window moves before the next batch is created. Keeping these two apart in your head is the key to everything that follows: changing one does not automatically change the other.

To make the movement concrete, consider a short sequence of ten tokens, numbered by position from 0 to 9. The window size in this trace is 4, so every batch contains four consecutive tokens. Positions are counted from zero, the same way a list is indexed in Python.

Read the output carefully. The first batch starts at position 0. The second starts at position 1, because the stride of 1 moved the window by one position. Each new batch begins exactly one position after the previous batch began, and each batch still holds four tokens. The window size never changed; only the starting point moved.

The diagram below shows the same ten positions and highlights the tokens covered by the second batch when the stride is 1. Positions 1 through 4 are inside the window at that step, which is why the second batch is 'model reads long text'.

Notice that the second batch is not a new window of text placed somewhere else. It is the same frame moved one position to the right. Positions 1, 2, and 3 were already inside the first batch, and position 4 is the one new token that enters the frame.

A stride of 1 produces heavy overlap between batches. Each batch shares most of its tokens with the batch before it, and only one new token enters per step. This is the behavior the source describes when it says the second batch repeats part of the first batch and adds one new word at the end.

The overlap is not a fixed property of the window. It is produced by the stride. Hold the window size at 4 and change only the stride, and the amount of shared text changes with it. The comparison below applies three different strides to the same ten tokens, with the same window size each time.

Read each column from top to bottom. In the stride 1 column, the second batch repeats three tokens from the first. In the stride 2 column, the second batch repeats only two. In the stride 4 column, the second batch shares nothing with the first, because the window jumped forward by four positions and each new batch begins exactly where the previous one ended. The window size was 4 in every column. Only the stride changed, and the overlap changed with it.

It helps to see batch creation as a fixed sequence of steps. The stride shift is applied between emitting one batch and emitting the next, so it controls where the next batch starts without changing how many items that batch contains.

The source notes that setting the stride equal to the window size can prevent overlap, because each new window begins exactly where the previous one ended. A stride larger than the window size goes one step further: the window jumps past positions that no batch has covered. Using the same ten tokens, a window size of 4, and a stride of 5, the first batch covers positions 0 through 3 and the second covers positions 5 through 8. The token at position 4, 'text', belongs to neither batch.