Tokenization and Byte Pair Encoding

A tokenizer's vocabulary can be extended with special context tokens that carry meanings ordinary words cannot express.

Programming

When a Word Is Not in the Vocabulary

Every tokenizer works from a fixed vocabulary, which is a table that assigns each known token an integer ID. The table is built before the tokenizer is used, from the material it was given. Real text is open-ended, though. Sooner or later a tokenizer will meet a word that was never part of that material. A new product name, a rare dialect form, or a misspelling can all appear at the moment of use.

What do you think happens?

The vocabulary has no entry for a word that appears in the input text. What should the tokenizer do: stop, discard the word, or emit something in its place?

Stop tokenizing and raise an error
Discard the word and continue
Emit a special marker token in place of the word and continue

Reveal answer

Answer:

Emit a special marker token in place of the word and continue

The source describes the tokenizer as modified so that an unknown word is replaced by an <|unk|> token, and tokenization keeps going without stopping or discarding the text.

Why Ordinary Words Are Not Enough

An ordinary vocabulary is built from the words and fragments found in the training data, and each one receives an ID. Those entries can only say what the text said. They cannot say that a word was unknown, or that one document has ended and another has begun. Some information about the text's structure therefore has no entry in an ordinary vocabulary at all.

The source handles this by extending the tokenizer with special context tokens. A special context token is an additional entry that does not come from the text itself. Its purpose is to carry a meaning that ordinary words cannot express, such as this word was unknown or a new document starts here. The tokenizer class used in the source's running example is extended in exactly this way, with two new tokens: one for unknown words and one for document boundaries.

Adding Special Tokens to a Vocabulary

Definition

A special context token is a vocabulary entry that does not come from the training text. It is given its own integer ID, placed alongside the existing IDs, so that the tokenizer can emit it like any other token.

The key point is that a special token is an ordinary vocabulary member once it has been added. It does not need separate handling inside the tokenizer's output. It has a string form and an integer ID, just like the words around it. The source's illustration shows a sample sentence tokenized against an existing vocabulary, after which the vocabulary is extended with two additional entries. The unknown-word token receives ID 783 and the document-boundary token receives ID 784 in that figure. Those particular numbers depend on how many entries the existing vocabulary already holds, so the lasting lesson is the mechanism, not the values.

Each special token string maps to its own integer ID, placed alongside the ordinary entries of the vocabulary.

How the Unknown-Word Fallback Works

The first special token, <|unk|>, is the tokenizer's fallback for unfamiliar words. The tokenizer is modified so that when it encounters a word that is not part of the vocabulary, it emits an <|unk|> token instead. The tokenizer does not need to stop or discard the text at that point. It substitutes the marker and continues with the next piece of input.

The tokenizer checks each incoming word against the vocabulary. A known word yields its own ID; an unknown word yields the <|unk|> ID, and tokenization then continues.

The marker is a deliberate trade. The original word cannot be recovered from the <|unk|> token itself, because every unknown word maps to the same ID. What survives is the signal that something was there. The model still receives a position in the token sequence marked as unfamiliar, instead of a silent gap or an error that halts processing.

In the real world

Suppose the vocabulary was built from older text and a user types a brand-new product name. Every ordinary piece of that name that the vocabulary does not contain is replaced with <|unk|>, while the surrounding familiar words keep their own IDs. The sequence stays intact in length and order, and the unfamiliar part is flagged rather than lost without trace.

Marking Boundaries with the End-of-Text Token

The second special token, <|endoftext|>, solves a different problem. A GPT-like language model is trained on many independent documents or books. Those texts are usually concatenated into one long stream of tokens for training. Without any marker, the model would see one sentence flow straight into an unrelated one, as though the two belonged to a single narrative.

The source notes that it is common to insert an <|endoftext|> token before each document that follows a previous text source. The token tells the model that the surrounding material, although joined together for training, does not form one continuous story. The boundary is marked explicitly in the token stream, so the model does not have to guess where one source stops and the next begins.

Preparing a Mixed Batch of Documents

Now trace how several unrelated documents become one training sequence. The source's arrangement places four independent passages in sequence: a sports result, a fairy tale, a financial report, and a romantic moment. Each passage after the first is preceded by an <|endoftext|> token, and the result is a single concatenated stream of tokens.

Each passage after the first is preceded by an <|endoftext|> token, so the boundary between unrelated sources is explicit in one continuous stream.

Worked example

Joining three independent documents

Three unrelated documents must become one training stream: document A is a short match report, document B is a short story, and document C is a short business note. Show where each <|endoftext|> token goes.

1
Tokenize each document separately
Each document is converted into its own sequence of token IDs using the same vocabulary, with any unfamiliar words replaced by <|unk|>.
2
Keep the first document as it is
The first source has no preceding text, so nothing is inserted before document A. The rule in the source places the token before each document that follows a previous text source.
3
Insert a separator before document B
Document B follows document A, so an <|endoftext|> token is placed between the end of A's tokens and the start of B's tokens.
4
Insert a separator before document C
Document C follows document B, so a second <|endoftext|> token is placed between B's tokens and C's tokens.
5
Concatenate in order
The three token sequences and the two separators are joined into one continuous stream: A tokens, then the separator, then B tokens, then the separator, then C tokens. The model sees each boundary explicitly and learns that the surrounding text is not one continuous narrative.

Result

One stream: [A tokens] <|endoftext|> [B tokens] <|endoftext|> [C tokens]. There are two separators for three documents, because the token precedes each document that follows another source.

Mistakes Beginners Make with Special Tokens

Common mistake
Assuming <|unk|> preserves the original word
Why it fails
Every unknown word maps to the same <|unk|> ID, so the specific word cannot be recovered from the token.
Do this instead
Treat <|unk|> as a flag that something unfamiliar was present, not as a stored copy of the word.
Common mistake
Expecting the tokenizer to stop when it meets an unknown word
Why it fails
The source describes the fallback as a substitution that lets tokenization continue.
Do this instead
Expect an <|unk|> token in the output at that position and continue reading the sequence.
Common mistake
Thinking special tokens are a separate kind of data
Why it fails
Once added, a special token receives its own ID and is emitted like any other vocabulary entry.
Do this instead
Handle <|unk|> and <|endoftext|> as ordinary vocabulary members whose meaning is structural.
Common mistake
Placing a separator between every pair of text and at the very beginning
Why it fails
The source inserts the token before each document that follows a previous text source, not before the first one.
Do this instead
Insert one separator between each pair of consecutive sources, and none before the first document.
Common mistake
Hard-coding the IDs 783 and 784 as universal values

Checking Your Understanding of Separators

EASY

A training corpus contains five independent articles, A through E, joined in that order. Using the rule that <|endoftext|> precedes each document that follows a previous source, how many separator tokens appear in the stream, and between which articles are they placed?

Hints

Count how many documents follow another source. The first article has no predecessor.
Each separator sits between two consecutive articles, so think of the gaps between A and B, B and C, and so on.

Your attempt (not saved)

MEDIUM

A tokenizer built from a small vocabulary receives the phrase 'quantum flux' and neither word is in the vocabulary. Write out the token sequence the tokenizer produces for those two words, using the names of the tokens rather than their IDs.