Creating Training Batches for Language Models

Every tokenizer works from a fixed vocabulary, a table that assigns each known token an integer ID. Real text is open-ended, though. Sooner or later a tokenizer meets a word that was never part of the material it was built from. A new product name, a rare dialect form, or a misspelling can all show up at the moment of use. A tokenizer that only knows its vocabulary has no answer for these words unless the vocabulary and the tokenizer are extended.

This article follows one idea through the whole process. The vocabulary can be extended with special context tokens, which handle unfamiliar words and also make clear where one unrelated piece of text stops and another begins. Those tokens then feed into the step that turns raw sequences into a training batch, which is handled by a custom collate function.

An ordinary vocabulary is built from the words and fragments found in the training data, and each one receives an ID. A special context token is an additional entry that does not come from the text itself. Its purpose is to carry a meaning that ordinary words cannot express, such as 'this word was unknown' or 'a new document starts here'.

Adding such an entry means giving it its own ID, placed alongside the existing ones. Once that is done, the tokenizer can emit the special token exactly like any other vocabulary member. The special tokens are not a separate mechanism; they are ordinary entries that happen to carry a meaning the training text itself cannot supply.

The lesson to keep from the illustration is that the special tokens are ordinary vocabulary members once they have been added. The specific numbers 783 and 784 are not general facts. A vocabulary that already holds a different number of entries would assign different IDs to the same two tokens.

The first special token, <|unk|>, is the tokenizer's fallback for unfamiliar words. The rule is direct: when the tokenizer encounters a word that is not part of the vocabulary, it emits an <|unk|> token instead. The tokenizer does not stop, and it does not discard the text at that point. It substitutes the marker and keeps going, so tokenization of the rest of the text can continue.

Because the marker occupies one position in the token sequence, the model still receives a signal that something was there. The original word cannot be recovered from the token itself. The model knows an unknown word occurred at that position, but not which word it was.

The second special token, <|endoftext|>, solves a different problem. When a GPT-like language model is trained on many independent documents or books, those texts are usually concatenated into one long stream of tokens. Without any marker, the model would see one sentence flow straight into an unrelated one. The <|endoftext|> token tells the model that the surrounding material, although joined together for training, does not belong to one continuous narrative.

The material notes that it is common to insert an <|endoftext|> token before each document that follows a previous text source. The same token can be read as the close of one document and the opening of the next, so the boundary is visible from either side of it.

The illustration above is a generated example that follows the described arrangement: an endoftext token precedes each document that follows an earlier text source. Document A has no token before it because nothing precedes it in the stream.

Training usually happens on groups of examples at once rather than one example at a time. That group is called a batch. Before a batch can be fed to a model, its contents must be arranged into a consistent form. The step that does this work is the custom collate function, which takes raw sequences and turns them into something a training run can use.

Consider a small batch of three sequences whose lengths are 5, 3, and 4 tokens. These lengths are invented for this illustration and are not taken from the material. Before the collate function does anything, the three sequences cannot sit together as a uniform block, because they have different lengths.

The material presents three responsibilities together as the work that turns raw sequences into a training batch. Each one answers a different need: making the sequences fit a common shape, supplying the target values the model is trained to predict, and marking which positions are only filler so they do not influence training.

Several details are outside what the material establishes. It does not give the procedure for creating target token IDs, so that step should be learned from a fuller description rather than inferred here. It does not name the filler value used for padding or explain how the common length is chosen. It does not give the specific IDs that a given tokenizer will assign to its special tokens, because those depend on how many entries the existing vocabulary already holds.

What the material does specify is clear: the unk token replaces a missing word so tokenization can continue, the endoftext token separates unrelated text, the collate function pads, creates targets, and masks padding, and masking marks positions rather than removing them.