Sliding Window Approach for Generating Input-Target Pairs

Before a language model can learn from a text, that text has to be converted into a form the training process can use. Earlier steps in this chapter covered how raw text is broken into tokens. This article picks up at that point. You already have a sequence of tokens, and the question is how to turn that sequence into the examples a model actually trains on.

A useful starting point is a small program that reads the full text of a short story called The Verdict, converts the raw string into a list of token IDs, and prints how many tokens the list contains. The length it prints is treated as the total number of tokens in the training set. Notice what this program leaves behind: a tokenized list, not the raw text. That tokenized list is what the data loader works with, and the rest of this article explains why.

An LLM learns by predicting the next word in a piece of text. Training does not hand the model a finished answer key. Instead, the training data is cut into pairs. One part is an input block, which is the portion of the text the model sees. The other part is a target, which is the word that comes immediately after that input block. The model's job is to predict the target from the input. Each time its prediction is compared against the actual next token, the model gets a learning signal.

Because every position in the text offers a new input and a new target, one piece of text can supply many training examples. The rest of the article shows how a sliding window generates those examples systematically.

Before any window moves, the source removes the first 50 tokens from the tokenized text. It does this only to make the passage used in later steps more interesting. The remaining tokens form the sequence the rest of the process works on. Think of the token list as a long row of discrete pieces, each with a fixed position.

The pairing has a second half that matters just as much. Words that come after the target are masked out during training, so the model cannot use them as hints. The model only ever sees the input block and is scored on whether it predicts the next word. The same token sequence can therefore be split into many different splits, and each split gives the model a different input and a different target.

The diagram shows one sample split three ways. The input block is the part the model reads. The target is the single token directly after it. Everything beyond the target stays hidden. The sliding window is simply a systematic way of producing many such splits from one long token sequence.

The source uses a sliding window approach to fetch input-target pairs from the training dataset. Picture a fixed-size frame placed over the token sequence. The frame covers an input block, and the token directly after the frame is the target. Each window position produces one input-target pair. Moving the frame one step along the sequence produces the next pair.

To make the mechanics easy to follow, consider an invented sequence of six tokens, named tok1 through tok6, with a window that holds three input tokens. The window always has the same width. The first window covers tok1, tok2, and tok3, and its target is tok4.

The window does not slide over raw characters or whole words on a page. It slides over tokens, which are the units the model already treats as discrete pieces. Tokenization here uses a BPE tokenizer, which splits text into subwords. The figure that shows input and target pairs leaves this step out on purpose, but the text must be tokenized before the LLM can process it.

Read the pipeline from left to right. The tokenized data is the source material, the sliding window is the mechanism that scans it, and the input-target pairs are the result that feeds LLM training. Tokenized data is the input to the process, and input-target pairs are its output. The source describes this procedure as the way a data loader fetches pairs from the training dataset, and the program that turns the file into a token list is the first stage of that loader.