Special Context Tokens

A language model does its arithmetic on numbers, not on words. Before any text can reach the model, each piece of it needs a numeric stand-in. Converting text into that numeric form is a two-part job. The first part is cutting text into smaller units called tokens. The second part is looking each token up in a structure called a vocabulary, which gives every distinct token a unique integer label. Once both are in place, converting new text is mostly a matter of looking things up.

This article follows that translation in both directions. It starts with text that is new to the system and expresses it as a sequence of token IDs. It then follows the reverse journey, turning numeric output back into readable words, because a model whose output cannot be read back into text is of little use. Along the way it introduces two special context tokens, <|unk|> and <|endoftext|>, that solve problems ordinary vocabulary entries cannot solve.

Tokenization is the first step whenever a new text sample arrives. The sample is cut into tokens in exactly the same manner that was applied to the training set, so its pieces can be compared with the pieces the vocabulary already knows. A token is therefore not a fixed property of language in general. It is whatever unit the tokenizer produces from the text in front of it. The examples in this article use a deliberately simple rule: split on spaces and keep only plain lowercase words. Real tokenizers can be considerably more sophisticated, but the lookup step works the same way regardless.

A vocabulary is a dictionary that associates each token with a unique integer label. Each token is the key and its integer is the value. No integer is shared by two words, and the vocabulary stores each token only once, even if a sentence uses the same word several times. The vocabulary is built from the entire training set, but once it exists it is not tied to that original set. It can be applied to the training set itself and also to new text samples.

The conversion so far runs from text toward numbers. A language model, however, produces numbers as its output, and those numbers must be turned back into text before a person can read them. This is the job of the inverse vocabulary. It is the same vocabulary read the other way: given a token ID, it returns the token that ID stands for. Applying this lookup to every ID in an output sequence, in order, reconstructs the text.

Every tokenizer works from a fixed vocabulary. Real text is open-ended, though, and sooner or later a tokenizer meets a word that was never part of the material it was built from. A new product name, a rare dialect form, or a misspelling can all appear at the moment of use. To handle such cases, a vocabulary can be extended with special context tokens. A special context token is an additional entry that does not come from the text itself. Its purpose is to carry a meaning that ordinary words cannot express, such as 'this word was unknown' or 'a new document starts here'.

Adding a special entry works the same way as adding any other entry: the new token is given its own integer ID, placed alongside the existing ones, so the tokenizer can emit it like any other token. The source's illustration assigns the unknown-word token the ID 783 and the document-boundary token the ID 784. Those specific numbers depend on how many entries the existing vocabulary already holds, so the lesson to keep is that once added, special tokens are ordinary vocabulary members.

The first special token, <|unk|>, is the tokenizer's fallback for unfamiliar words. The tokenizer is modified so that when it encounters a word that is not part of the vocabulary, it emits an <|unk|> token instead. The tokenizer does not need to stop or discard the text at that point. It substitutes the marker and keeps going, so tokenization can continue through the rest of the sample.

The marker occupies one position in the token sequence. Even though the original word cannot be recovered from the token itself, the model still receives a signal that something was there. This is the core trade-off: the unknown token preserves the position and the fact of an unfamiliar word, but not its spelling.

The second special token, <|endoftext|>, solves a different problem. When a GPT-like language model is trained on many independent documents or books, those texts are usually concatenated into one long stream of tokens. Without a marker, the model would see one sentence flow straight into an unrelated one. The common practice is to insert an <|endoftext|> token before each document that follows a previous text source. The token tells the model that the surrounding material, though joined together for training, does not belong to one continuous narrative.

The timeline shows the arrangement. Four independent passages, covering a sports result, a fairy tale, a financial report, and a romantic moment, are placed in sequence. Each passage after the first is preceded by an <|endoftext|> token, producing a single concatenated stream. The first passage has no marker in front of it. The same token plays a dual role: it closes one document and opens the next, so the boundary is visible from either side.

Putting the pieces together, a set of independent documents becomes one prepared training input through a sequence of steps. Each document is tokenized using the same cutting rule, each token is looked up in the vocabulary, the separator token is inserted before every document that follows another text source, and the results are concatenated into one stream of token IDs.