Tokenization and Building a Vocabulary

A tokenizer converts text into integer IDs by looking up each token in a vocabulary. That vocabulary is a fixed table built from the text the tokenizer was given. The moment the text contains a word that is not in that table, a problem appears: a plain lookup has no entry to return for that word, so the conversion cannot finish cleanly unless the tokenizer has a rule for this case.

This article follows one answer to that problem. The class SimpleTokenizerV2 is a modified version of the simpler tokenizer that came before it. Where the earlier version could not account for a word outside its vocabulary, the new version replaces unknown words with a special placeholder, written <|unk|>, which stands for unknown. Your job is to follow exactly how that replacement happens, and to notice what the replacement costs.

Encoding a sentence has four stages. First, a regular expression splits the text on punctuation and whitespace. Second, empty pieces are removed and surrounding spaces are stripped. Third, every piece is checked against the vocabulary, and any piece that is not present becomes <|unk|>. Fourth, each remaining item is looked up in the vocabulary to produce its integer ID. The replacement happens in the third stage, after the text has been split and cleaned but before the IDs are looked up.

Most of the change is contained in one line inside encode. That line checks each item against the vocabulary and substitutes the placeholder when the check fails. The whole behavior depends on that conditional expression, so it is worth reading slowly. The line that follows it converts every item into an ID with a list comprehension over the replaced items.

The ID lookup runs on the replaced items too. Therefore the placeholder must itself be a key in the vocabulary. The idea behind the fix is simple: instead of failing on an unfamiliar word, the tokenizer substitutes a placeholder that is already in the table, and that placeholder has its own integer ID. Every token in the sentence, including the unfamiliar ones, then ends up with an ID the rest of the pipeline can handle.

Think of the vocabulary as a table where each row pairs a token string with its integer ID. In the small example used in this article, the rows are hello paired with 0, world paired with 1, and <|unk|> paired with 2. Encoding is the act of finding each item's row and reading off its ID. An item with no row is the case the tokenizer must handle, and the placeholder row is the answer it falls back on.

Use the same procedure on a different input. Suppose the vocabulary is the same: hello is 0, world is 1, and <|unk|> is 2. Encode the sentence 'world there hello'. Split it, clean it, replace the unknown word, and then look up each item. The order of the output follows the order of the input words, so the unknown word's ID appears in the middle position, between the IDs of its neighbors.

Decoding reverses the lookup. The decode method maps each ID back to its string through the reverse dictionary, joins the strings with spaces, and then removes the spaces that appear just before certain punctuation marks. Decoding the list [0, 2, 1] therefore produces the text 'hello <|unk|> world'. The word there does not come back.

The loss happens in the replacement stage of encode, not in decode. Once there is swapped for <|unk|>, no later step has the original text available to restore. Decode only sees IDs, and the ID for the placeholder maps to the placeholder string, so the round trip returns a sentence that differs from the input at exactly the unknown position.