Batch Size and Its Effect on Training
Window size and stride are two independent settings. Changing the stride changes the overlap between batches, while the window size stays the same.
Why Long Text Is Split Into Batches
When a language model is trained on a long piece of text, the whole text cannot be fed in all at once. It has to be divided into smaller pieces called batches, and each batch is built from a portion of the input dataset. The way those pieces are chosen determines which words the model sees together and how much of the text repeats from one piece to the next. The setting that controls this behavior is called stride.
This article answers one focused question: when the next batch is created, how far does the reading position move? Once you can answer that, you can predict how much two neighboring batches share.
The Sliding Frame Picture
Picture a small frame placed over a passage of text. You read only what falls inside the frame, then slide the frame forward to read the next portion. Each position of the frame produces one batch: the items visible inside it at that moment.
The frame has a fixed size, and the only thing that varies from one step to the next is how far it slides. Keep this picture in mind, because every stride question in this article reduces to asking how far the frame moved.
Stride and Input Window Defined
Stride is the number of positions by which the input window shifts when the next batch is created from an input dataset.
The input window is the frame described above. Its size sets how many items a batch contains. The stride is separate: it sets how far the window moves before the next batch is created. A stride of $n$ moves the window by $n$ positions, and that movement determines how much consecutive batches overlap.
Window size and stride are two independent settings. Changing the stride changes the overlap between batches, while the window size stays the same.
The two extreme cases named in the source are a stride of 1, which shifts the window by one position each time and produces heavy overlap between batches, and a stride equal to the input window size, which can prevent overlap because each new window begins exactly where the previous one ended.
Tracing a Window Across Eight Tokens
To make the movement concrete, consider a short token sequence of eight items and an input window that holds four tokens. Positions are numbered from 0. The first batch always covers positions 0 through 3. The diagram below shows the window at that first step.
Trace batches with a stride of 1
Sequence: the cat sat on the mat and slept (positions 0 to 7). Window size is 4. Stride is 1. List the first three batches.
- 1Batch 1
The window starts at position 0 and covers positions 0 to 3: the cat sat on.
- 2Shift the window
Stride is 1, so the window moves forward by one position, to start at position 1.
- 3Batch 2
The window now covers positions 1 to 4: cat sat on the. Three of its four tokens were already in batch 1.
- 4Shift again
The window moves forward by one more position, to start at position 2.
- 5Batch 3
The window covers positions 2 to 5: sat on the mat. Two of its tokens, sat and on, were already in batch 1.
Batch 1: the cat sat on. Batch 2: cat sat on the. Batch 3: sat on the mat. Each batch has four tokens, and each new batch repeats most of the previous one.
Batch Sequence as the Window Advances
Batches are produced in order, and each one follows from the previous one by the same shift. The diagram shows the first three batches under a stride of 1. Each arrow is one application of the stride, and the batch after it begins one position later than the batch before it.
Comparing Strides Side by Side
The same window size with different strides produces different overlap. The diagram below applies three strides to the same eight-token sequence with a window of four. Only the shift differs between the columns.
Read each column from top to bottom. Under a stride of 1, the second batch repeats most of the first and adds one new token at the end. Under a stride of 4, which equals the window size, the second batch jumps to the next unread block and shares no tokens with the first. The stride changed the overlap, and the window size stayed at four in every column.
| Stride | Window moves by | Batch 2 tokens | Tokens shared with batch 1 | Overlap level |
|---|---|---|---|---|
| 1 | 1 position | cat sat on the | cat, sat, on (3 tokens) | Heavy overlap |
| 2 | 2 positions | sat on the mat | sat, on (2 tokens) | Partial overlap |
| 4 | 4 positions | the mat and slept | none | No overlap |
Three strides applied to the same sequence with a window of four tokens
When the Stride Is Larger Than the Window
A stride equal to the window size is the boundary case where batches meet end to end without sharing tokens. Setting the stride above the window size goes one step further, and this is where tokens can be missed entirely. Using the same sequence and a window of four, a stride of 5 places batch 1 at positions 0 to 3 and batch 2 at positions 5 to 8. Position 4, the token the, is covered by neither batch.
Predicting Overlap Before Revealing the Answer
What do you think happens?
Using the same sequence (the cat sat on the mat and slept) and a window of four, which tokens appear in both batch 1 and batch 2 when the stride is 2?
Reveal answer
sat and on
Batch 1 covers positions 0 to 3 and batch 2 covers positions 2 to 5. The overlapping positions are 2 and 3, which hold sat and on.
Keep the same sequence and window size of four. Predict the first two batches for a stride of 3. How many tokens do they share, and is any token skipped? Check your answer by counting positions in batch 1 and batch 2.
Hints
- Batch 1 always covers positions 0 to 3.
- With stride 3, batch 2 starts at position 3, so it covers positions 3 to 6.
- Position 3 is shared, and check whether any position from 0 to 6 is missing from both batches.
Common Misunderstandings About Stride
Common mistake Treating stride and window size as the same setting
What it looks likeAssuming that a window of four with a stride of four is the same as a window of two with a stride of two.
Why it failsThe window size sets how many items a batch contains, while the stride sets how far the window moves. They are separate settings.
Do this insteadRead the window size and the stride as two independent numbers. Changing one does not automatically change the other.
Common mistake Believing a smaller stride always produces better batches
What it looks likeChoosing a stride of 1 because it shows every position, without considering how much of each batch repeats.
Why it failsA stride of 1 produces heavy overlap, so consecutive batches repeat most of their content.
Do this insteadChoose the stride by deciding how much repetition between batches is acceptable for the training setup.
Common mistake Assuming a stride equal to the window size leaves gaps
What it looks likeThinking that when the stride equals four and the window holds four tokens, some tokens must be missed.
Why it failsA stride equal to the window size places each new window exactly where the previous one ended, so no tokens are skipped and no tokens overlap.
Do this insteadRemember that gaps appear only when the stride is larger than the window size.
Common mistake Assuming all tokens always appear in some batch
What it looks likeExpecting the token at position 4 to appear in a batch when the stride is 5 and the window is 4.
Why it failsA stride larger than the window moves past positions, so those positions are never inside any window.
Do this insteadCompare the stride with the window size before assuming full coverage of the dataset.
Key Takeaways
- Stride is the number of positions by which the input window shifts when the next batch is created from an input dataset.
- The input window size sets how many items a batch contains, and the stride sets how far the window moves; these are separate settings.
- A stride of 1 produces heavy overlap between consecutive batches, while a stride equal to the window size can prevent overlap.
- A stride larger than the window size leaves some positions covered by no batch, so tokens can be skipped entirely.