Building a Vocabulary from Training Text

Every tokenizer works from a fixed vocabulary, which is a table that assigns each known token an integer ID. The tokenizer looks up each piece of incoming text in that table and emits the matching ID. The difficulty is that real text is open-ended. A new product name, a rare dialect form, or a misspelling can appear at the exact moment the tokenizer is being used, even if it never appeared in the material the vocabulary was built from.

This article shows how a vocabulary can be extended with special context tokens. These tokens handle unfamiliar words and also make clear where one unrelated piece of text stops and another begins.

An ordinary vocabulary is built from the words and fragments found in the training data, and each one receives an ID. A special context token is an additional entry that does not come from the text itself. Its job is to carry a meaning that ordinary words cannot express, such as 'this word was unknown' or 'a new document starts here'.

Adding a special token means giving it its own ID, placed alongside the existing ones. Once that is done, the tokenizer can emit it exactly like any other token. The source illustrates this with a vocabulary that is extended by two entries: 783 is assigned to the unknown-word token and 784 to the document-boundary token. Those particular numbers depend on how many entries the existing vocabulary already holds, so the lesson to keep is that the special tokens are ordinary vocabulary members once they have been added.

The first special token, <|unk|>, is the tokenizer's fallback for unfamiliar words. The source describes the rule directly: the tokenizer is modified so that when it encounters a word that is not part of the vocabulary, it emits an <|unk|> token instead. The tokenizer does not stop, and it does not throw the text away at that point. It substitutes the marker and continues with the next word.

The marker matters even though it hides the original word. The <|unk|> token occupies one position in the token sequence, so the model still receives a signal that something was there. The original word, however, cannot be recovered from the token itself. The token records the event of an unknown word, not its spelling.

The second special token, <|endoftext|>, solves a different problem. When a GPT-like language model is trained on many independent documents or books, those texts are usually concatenated into one long stream of tokens. Without any marker, the model would see one sentence flow straight into an unrelated one. The source notes that it is common to insert an <|endoftext|> token before each document that follows a previous text source. The token tells the model that the surrounding material, though joined for training, does not belong to one continuous narrative.

The source's illustration uses four independent passages: a sports result, a fairy tale, a financial report, and a romantic moment. They are placed in sequence, and each passage after the first is preceded by an <|endoftext|> token. The result is one concatenated stream. The source caption describes these tokens as markers that signal the start or end of a segment. The same token can be read as the close of one document and the opening of the next, so the boundary is visible from either side.