Tokenization using a BPE Tokenizer
Tokenization is therefore the step that defines the units the sliding window moves over.
Starting From a Token Count
This article begins with a short program rather than a definition. The program reads the full text of a short story called The Verdict, converts the raw string into a list of token IDs, and prints the length of that list. The source treats this length as the total number of tokens in the training set.
Before reading on, try to form a picture of what the program leaves behind. It creates a raw string, then a list of integers, and it prints only the length of that list. Ask yourself which of those values is the one the rest of the article depends on. The explanation that follows argues that the tokenized list, not the raw text, is what the data loader works with.
The source then removes the first 50 tokens from the tokenized text, but only to make the passage used in later steps more interesting. Make a prediction about what that operation does to the list before you go on, and check it against the trace in the sections below.
Why Next-Word Prediction Supplies the Training Signal
An LLM learns by predicting the next word in a piece of text. Training does not hand the model a finished answer key. Instead, the training data is cut into pairs. Because the text itself supplies both halves of each pair, the model can be trained on ordinary text without anyone writing separate labels.
Input block: the portion of the text the model is allowed to see. Target: the word that comes immediately after the input block, which the model must predict from the input.
The training signal comes from comparing the model's guess against the target. The model reads the input block, produces a prediction for the next word, and is scored on whether that prediction matches the target. Each such comparison is one lesson the model receives from the text.
Splitting a Sample Into Input, Target, and Masked Words
The pairing has a second half that matters just as much as the first. Words that come after the target are masked out during training, so the model cannot use them as hints. The model only ever sees the input block and is scored on whether it predicts the next word.
Because of this split, one piece of text can be turned into many training examples. Each new subsample gives the model a different input and a different target. The diagram below uses a generated six-token sequence to show one sample divided into its three parts: the input block the model may read, the single target it must predict, and the hidden tokens beyond the target.
Read the diagram from left to right. The first three tokens form the input block and are the only tokens the model may read. The fourth token is the target. The last two tokens are masked, meaning they exist in the text but are kept out of what the model sees for this example.
Tracing a Sliding Window Across Tokens
The source uses a sliding window to fetch input-target pairs from the training dataset. Picture a fixed-size frame placed over the token sequence. The frame covers the input block, and the token directly after the frame is the target. Each window position produces one input-target pair. Moving the frame along the sequence produces the next pair.
Sliding a width-3 window over six tokens
The tokens are Every, effort, moves, you, forward, and a period. The window holds three input tokens. Trace the input-target pairs produced as the window moves one token at a time.
- 1Window at the start
The frame covers Every, effort, moves. The token directly after the frame is you, so the first pair is input Every effort moves with target you.
- 2Shift forward by one
The frame moves one token to the right and now covers effort, moves, you. The token after the frame is forward, so the second pair is input effort moves you with target forward.
- 3Shift forward again
The frame now covers moves, you, forward. The token after it is the period, so the third pair is input moves you forward with target period.
- 4Stop at the end
No further full-width frame fits with a token after it, so the window stops. Three pairs were produced from six tokens.
Three input-target pairs: (Every effort moves, you), (effort moves you, forward), and (moves you forward, period). The window width stays at three throughout, while both the input block and the target move forward by one token.
Notice what changes between the windows. The input block shifts forward by one token, and so does the target. The model is asked to predict a different next word each time, while the window keeps the same width. The data loader performs this procedure to fetch pairs from the training dataset.
['Every', 'effort', 'moves'] -> you
['effort', 'moves', 'you'] -> forward
['moves', 'you', 'forward'] -> .- The list holds the generated token sequence used throughout this section, not the tokens of the source text.
- The slice tokens[i:i + window] takes the input block, and tokens[i + window] takes the token directly after it.
What do you think happens?
If the window width were 2 instead of 3 over the same six tokens, what would the first input-target pair be?
Reveal answer
The input block would be Every effort, and the target would be moves.
The window covers two tokens, and the token directly after the frame becomes the target. The width sets how many tokens the model reads, and the target is always the token that follows the frame.
Mistakes Beginners Make With Input-Target Pairs
Common mistake Treating the raw text as the training data
Why it failsThe windows move over a token sequence, not over the raw string. The text has to be converted into tokens with a BPE tokenizer before any pair can be formed.
Do this insteadThink of tokenization as the first stage of the data loader. Only the list of token IDs is passed on to the window.
Common mistake Letting the model see the words after the target
Why it failsWords past the target are masked out during training. If they were visible, the model could copy hints instead of predicting the next word.
Do this insteadKeep the three-way split in mind: input block visible, target to predict, everything beyond the target hidden.
Common mistake Thinking the target is part of the input block
Why it failsThe target is the single token immediately after the frame. It is what the model must predict, so it is not something the model reads.
Do this insteadWhen tracing a window, count the input tokens first, then identify the next token as the target.
Common mistake Assuming the window moves by more than one token each step
Why it failsEach new window produces a new pair by shifting one position, so the input block and the target move forward together by one token.
Do this insteadTrace each position in order, shifting the frame by a single token each time.
Practice Tracing a Window
Using the generated six-token sequence Every, effort, moves, you, forward, and a period, with a window width of three, list the input block and target for the window that starts at the second token. Then state which tokens are masked for that example.
Hints
- The second window starts at effort, so its input block covers effort, moves, you.
- The target is the token directly after the input block.
- Every token after the target is masked for that example.
Key Takeaways
- An LLM is pretrained by predicting the next word in a text, so its training data is cut into input-target pairs.
- The input block is what the model reads, the target is the next word it must predict, and words past the target are masked out during training.
- Text must be converted into tokens with a BPE tokenizer before pairs are formed, so the windows move over a token sequence rather than raw text.
- A sliding window of fixed width produces one input-target pair per position by shifting one token at a time, and the data loader uses this procedure to fetch pairs from the training dataset.