Building a Vocabulary from Text
A tokenizer class implements encode and decode, and it holds a vocabulary that maps text tokens to integer IDs along with an inverse mapping built from that vocabulary.
Starting From a Running Program
Most discussions of tokenization start with the theory of how text becomes numbers. This article starts with a running program instead. A small Python tokenizer class takes a sentence, turns it into a list of integers, and then tries to turn those integers back into text. Watching what the code does first gives every later explanation something concrete to attach to.
Keep one question in mind while you follow the trace: after the round trip from text to IDs and back, does the result still look like the input? Some of the answer will be surprising, and that surprise is useful. It points directly at the part of the implementation that deserves a close look.
What a Tokenizer Class Holds
A tokenizer, in the sense used here, is a class that implements two methods, encode and decode. The class holds a vocabulary, which is a mapping from text tokens to integer token IDs. Encode goes from text toward IDs. Decode goes from IDs back toward text.
The vocabulary is stored as a dictionary named str_to_int, which maps each text token to its integer. The class also keeps a second dictionary, int_to_str, built by swapping each key and value of the original vocabulary. The reverse mapping is built once, when the tokenizer is created, so that both directions have a lookup table ready to use. Decode never searches the original vocabulary backwards; it simply reads the inverse table.
The four-entry vocabulary in the diagram was written for this article so the steps are easy to follow. The left table answers the question text to ID. The right table answers the question ID to text. Both tables describe the same pairs, read in opposite directions.
Tracing Encode on a Short Sentence
Before reading the rules in detail, trace one input from start to finish. Use the four-entry vocabulary above, which holds the, cat, sat, and the period. Pass the string 'the cat sat.' to encode and follow what happens to it.
Encoding 'the cat sat.' with the four-entry vocabulary
Vocabulary: the maps to 0, cat maps to 1, sat maps to 2, and the period maps to 3. Encode the string 'the cat sat.' and produce a list of IDs.
- 1Split the string
The split pattern treats whitespace and punctuation as boundaries. Because the pattern is wrapped in a capture group, the boundary characters are kept as pieces. The string becomes the pieces the, a space, cat, a space, sat, and the period.
- 2Strip each piece
Surrounding whitespace is removed from every piece. The two space pieces become empty strings.
- 3Drop empty pieces
Any piece that is empty after stripping is removed. The two empty strings disappear, leaving the, cat, sat, and the period.
- 4Look up each piece
Each remaining piece is looked up in
str_to_int. The piece the gives 0, cat gives 1, sat gives 2, and the period gives 3. - 5Collect the IDs
The integers are gathered into a list in the same order as the pieces that produced them.
The list of IDs is 0, 1, 2, 3.
How Encode Builds the IDs
Encode works in three moves, and each move hands its output to the next. The first move splits the input string into pieces. The split pattern treats punctuation marks, the double-hyphen sequence, and whitespace as boundaries. Since the pattern is wrapped in a capture group, the boundary characters themselves are kept as pieces rather than thrown away. This is why the period in the example survives as its own token.
The second move strips surrounding whitespace from each piece and removes any piece that ends up empty. Splitting on whitespace produces pieces that are just spaces, and this move exists to discard them. The third move looks each remaining piece up in str_to_int and collects the resulting integers into a list.
- Move one: split the string at punctuation, double hyphens, and whitespace, keeping the boundary characters as pieces.
- Move two: strip whitespace from each piece and drop any piece that is empty.
- Move three: look up each remaining piece in
str_to_intand collect the integers in order.
Decoding the IDs Back Into Text
Decode reverses the lookup. It takes the list of IDs, looks up each ID in int_to_str to recover a text piece, joins the pieces with single spaces, and then applies one cleanup step that removes the space placed before certain punctuation marks. Feeding the list 0, 1, 2, 3 into decode with the same vocabulary gives a concrete case to check.
What do you think happens?
Before you read on, predict the output of decode when it receives the list 0, 1, 2, 3. Will it match the string the cat sat. exactly?
Reveal answer
It returns the cat sat.
The IDs map to the, cat, sat, and the period. Joining them with single spaces gives the cat sat . and the cleanup step removes the space before the period, leaving the cat sat. Here the output matches the input, because the only spacing decode had to rebuild was the space before the punctuation mark.
Where the Spacing Can Change
The round trip can fail to reproduce the original text in a narrow, specific way. The source passage used for this article is a longer excerpt from Edith Wharton's short story, containing double quotation marks, an apostrophe, a comma, and a period. When it is encoded and then decoded, all the words come back and the commas and periods stay attached to the words they follow. The mismatch lies in spacing. The output has a space after the opening double quotation mark, and the apostrophe in the word It's has a space after it.
The cause is the join and cleanup step, not the lookup. Decode joins every piece with a single space, and its cleanup only removes spaces in front of a fixed set of punctuation marks. Spaces after an opening quotation mark or after an apostrophe are not removed by that rule, so they survive into the output. The recovered text therefore differs from the original in spacing even when every token came back correctly.
Tokens Missing From the Vocabulary
A piece that has no entry in str_to_int has no ID to return, so encode cannot produce one for it. The vocabulary is therefore the set of pieces the tokenizer can represent. The source material does not describe a fallback value or substitute token for missing pieces, so none should be assumed. Whatever the implementation does with a missing piece is determined by the code that calls the lookup, and it changes the IDs that come out.
Case matters here too. The vocabulary is case-sensitive, so a word must appear in the dictionary exactly as it appears in the input. A capital letter in the input can therefore produce a missing piece even when the lowercase word is in the vocabulary.
Common Mistakes With Tokenizer Round Trips
Common mistake Assuming decode rebuilds the text exactly as it was typed
What it looks likeEncoding a passage with an opening double quotation mark and then decoding it yields a space after that quotation mark.
Why it failsDecode rebuilds spacing with a join and a limited cleanup step, so spacing is not guaranteed to match the original.
Do this insteadJudge a round trip by whether the right tokens came back, and check spacing separately.
Common mistake Thinking the inverse table is searched during decode
What it looks likeLooking for the text piece for ID 2 by scanning str_to_int for the value 2.
Why it failsThe class builds
int_to_stronce, when it is created, and decode reads that table directly.Do this insteadRead the inverse table int_to_str, which is the swapped copy of the forward vocabulary.
Common mistake Expecting encode to skip unknown words silently
What it looks likeEncoding a word with a capital letter when only the lowercase form is in the vocabulary.
Why it failsThe lookup is a plain dictionary access, and the vocabulary is case-sensitive. A missing piece has no ID to return.
Do this insteadCheck that every piece appears in the vocabulary exactly as written, including capitalization.
Common mistake Forgetting that boundary characters such as punctuation are kept as their own pieces
Trace It Yourself
Using the same four-entry vocabulary (the, cat, sat, period), trace encode on the string 'The cat sat.' Follow the three moves in order. Identify the first piece that cannot be looked up, and explain what encode has to return for it.
Hints
- Start by splitting the string at whitespace and the period, keeping the boundary characters.
- Compare each piece against the vocabulary exactly as written. Pay attention to capitalization.
- Once you find the missing piece, check what the lookup step has available to return for it.
Key Takeaways
- A tokenizer class implements encode and decode and holds a vocabulary mapping text tokens to integer IDs, plus an inverse mapping built from that vocabulary when the class is created.
- Encode splits the string at punctuation, double hyphens, and whitespace, strips each piece, drops empty pieces, and looks each remaining piece up to collect a list of IDs.
- Decode maps IDs through the inverse vocabulary, joins the tokens with single spaces, and removes spaces before a fixed set of punctuation marks.
- The round trip can change spacing, such as the space after an opening double quotation mark or after an apostrophe, because the join and cleanup steps cannot restore the original spacing.
- A piece missing from the vocabulary has no ID to return, and the vocabulary is case-sensitive.