Encoding and Decoding Text with a Tokenizer

A tokenizer turns text into integer IDs by looking up each token in a vocabulary. The vocabulary is a fixed table built from the text the tokenizer was given. That design raises an immediate problem. If a sentence contains a word that is not in the table, a plain lookup has no entry to return for that word, and the conversion cannot finish cleanly unless the tokenizer has a rule for this case.

This article follows one answer to that problem. The class SimpleTokenizerV2 is a modified version of the simpler tokenizer that came before it. Where the earlier version could not account for a word outside its vocabulary, the new version replaces each unknown word with a special placeholder written as <|unk|>, which stands for unknown. Your job is to follow exactly how that replacement happens, and to notice what it costs.

Think of the vocabulary as a small dictionary that pairs each known token with a number. Encoding is the act of looking each token up in that dictionary and writing down its number. Decoding is the reverse: a second table maps each number back to its token. Both directions depend on the vocabulary being fixed, so a token that was never added to it has no number to look up.

This three-entry vocabulary is the working example for the rest of the article. The word there is deliberately absent from it. Notice that the placeholder <|unk|> sits in the same table as the real words, with its own ID. That placement is what makes the replacement work.

Encoding a sentence has four stages. First, a regular expression splits the text on punctuation and whitespace. Second, empty pieces are removed and surrounding spaces are stripped. Third, every piece is checked against the vocabulary, and any piece that is not present becomes <|unk|>. Fourth, each remaining item is looked up in the vocabulary to produce its integer ID. The flowchart below traces these stages for a single word, and the rest of this section runs them on a full sentence.

Now take the sentence hello there world. Stage one splits it on whitespace into three pieces: hello, there, and world. Stage two finds no empty pieces and no extra spaces to strip, so the list is unchanged. Stage three checks each piece. The words hello and world are in the vocabulary and stay as they are. The word there is not, so it is replaced by <|unk|>. After this stage the list reads hello, <|unk|>, world. The original text there no longer exists anywhere in the list.

Most of the replacement behavior sits inside a single line in the encode method. That line checks each item against the vocabulary and substitutes the placeholder when the check fails. The whole behavior depends on that conditional expression, so it is worth reading slowly. The line that comes right after it does the lookup:

After replacement, the list no longer holds the original words. It holds a mix of real tokens and the placeholder, and the lookup treats them the same way. The diagram below shows the final position-by-position mapping for hello there world. Each position receives exactly one ID, and the middle position receives the ID of the placeholder rather than any ID tied to the word there.

Read the map from left to right. The first and last positions keep their words and therefore keep their own IDs. The middle position has lost the word there and holds the placeholder, so it carries the ID for <|unk|>. From the pipeline's point of view, the three positions now look like three ordinary tokens, and the rest of the code can process them without any special case.

The decode method reverses the lookup. It maps each ID back to its string through the reverse dictionary, joins the strings with spaces, and then removes the spaces that appear just before certain punctuation marks. Decoding the list 0, 2, 1 therefore produces the text hello <|unk|> world. The word there does not come back.