Tokenization for LLMs

A large language model learns from large amounts of text. However, the text in its raw form is not yet ready for training. Preparing text for large language model training is a required step before the model can learn from it. The model does not simply consume text as it was originally written; the text has to be transformed first.

This article focuses on the part of that preparation that comes right after tokenization. Tokenization is the process of splitting text into smaller units called tokens. Earlier steps cover how raw text is broken into tokens, so here we pick up at the point where you already have a sequence of tokens. The question is how to turn that sequence into the examples a model actually trains on.

The process described in the source has a clear input and a clear output. Tokenized data is the input. Input-target pairs are the output. Everything in between is the sliding window approach applied to the tokens.

The window does not slide over raw characters. It slides over the tokens. This is why tokenization comes first: the window operates on units the model already treats as discrete pieces. If the splitting step produced different tokens, the windows and the pairs built from them would also be different.

An input-target pair is a training example. Each pair ties a piece of input to the target the model should learn to associate with it. The source defines pairs by their purpose, which is to supply training examples, rather than by one fixed recipe. The exact arrangement can vary, so the illustration below shows one possible arrangement and says so clearly.

The sliding window approach moves a window across the tokenized data to produce these pairs. The window covers a fixed number of consecutive tokens. At each position it yields a pair, and then it moves forward. Because consecutive window positions overlap, one token sequence can yield several pairs.

The diagram above shows the window at its first position. Tokens The, cat, and sat form the input. Tokens cat, sat, and on form the target. Tokens cat and sat appear in both, which is the overlap described above. The tokens the and mat are not reached yet.

Now trace exactly what changes when the window advances by one position. Compare the state before the step with the state after it. Only the window contents, and therefore the generated pair, change. The token sequence itself stays the same.

Read the diagram as a before and after pair. Before the step, the window starts at position 0, its input is The cat sat, and its target is cat sat on. After one step, the window starts at position 1, its input is cat sat on, and its target is sat on the. The pair changes from one example to the next, while the six-token sequence it reads from stays unchanged.

The source describes tokenization as splitting text into word tokens and subword tokens. A word-level split can keep a word as one token. A subword split can produce several smaller tokens for the same word. The splitting decision changes what the tokenized data looks like, and that in turn changes every pair generated from it.

The following example is an illustration written for this article, not taken from the source. It uses the made-up word unhappiness. Under a word-level split, the whole word stays a single token. Under a subword split, the same word becomes several smaller pieces. The exact subword boundaries depend on the tokenizer used, so the split shown here is one plausible example.

Byte pair encoding, often abbreviated BPE, is described in the source as a more advanced way of tokenizing text. It is one method for producing tokens, and it belongs to the same tokenization step covered above. Its result feeds the same sliding window process that produces input-target pairs.

The source does not walk through BPE step by step, so this article does not state its internal rules. What matters at this stage is the change in state it produces. The text starts as a sequence of smaller units and ends as a sequence of tokens that the tokenizer has formed. Follow that change closely, comparing each state with the one before it, to see how the token sequence takes shape.