Regular Expressions for Text Splitting
A tokenizer class implements encode and decode, and it holds a vocabulary that maps text tokens to integer IDs along with an inverse mapping built from that vocabulary.
Watching a Tokenizer Run
Most explanations of tokenizers start with the theory of how text becomes numbers. This article starts with a running program instead. A small Python tokenizer class takes a passage of text, turns it into a list of integers, and then tries to turn those integers back into text. Watching that round trip first gives the explanation of each method something concrete to attach to.
What do you think happens?
After text is encoded into IDs and then decoded back, will the result look exactly like the input?
Reveal answer
The words come back, but spacing around some quotation marks and apostrophes can change.
Decode rebuilds spacing with a join and a cleanup step, so the recovered text can differ from the original in spacing. The words and the commas and periods are still present and attached to the words they follow.
Keep one question in mind as you read: after the round trip from text to IDs and back, does the result still look like the input? The surprising part of the answer points directly at the step of the implementation that deserves a close look.
Text Tokens and Integer IDs
A tokenizer, in the sense used here, is a class that implements two methods: encode and decode. The class holds a vocabulary, which is a mapping from text tokens to integer token IDs. Encode goes from text toward IDs. Decode goes from IDs back toward text.
The vocabulary is the central piece of data. Each token string corresponds to exactly one integer ID. The vocabulary is stored in a dictionary called str_to_int, which goes from token string to ID. The class does not search this dictionary backwards when it needs to go the other way. Instead, it builds a second dictionary, int_to_str, once, when the class is created, by swapping each key and value of the original vocabulary. That gives both directions a lookup table that is ready to use.
Encode, Step by Step
Encode works in three moves. Each move takes the output of the one before it. The diagram shows the whole path from the input string to the final list of IDs, and the subsections below describe each stage in turn.
Tracing a Small Encode Call
Encoding The cat, sat.
Using the illustrative vocabulary {The: 0, cat: 1, ',': 2, sat: 3, '.': 4}, trace encode on the string “The cat, sat.” and produce the list of IDs.
- 1Split
Punctuation and whitespace are boundaries and are kept as pieces. The pieces are The, a space, cat, a space, comma, a space, sat, period. Note that the comma and period appear as their own pieces.
- 2Strip and drop
Each piece is stripped. The space pieces become empty strings and are dropped. The remaining pieces are The, cat, comma, sat, period.
- 3Look up
The Dictionary lookup gives The to 0, cat to 1, comma to 2, sat to 3, and period to 4. Each piece exists in the vocabulary, so every lookup succeeds.
- 4Collect
The integers are collected into a list in the original order.
[0, 1, 2, 3, 4]
The same three moves run on the longer passage from Edith Wharton’s short story in the source. That passage includes double quotation marks, an apostrophe, a comma, and a period, which gives the splitting pattern several different kinds of boundary to handle. The small example here is easier to trace by hand, and the steps are identical.
Decode and the Inverse Vocabulary
Decode reverses the lookup. It takes a list of IDs and maps each one through the inverse dictionary int_to_str to recover a text piece. The pieces are then joined with single spaces. Finally, one cleanup step removes the space placed before a fixed set of punctuation marks. Only the join and the cleanup can change the spacing, and the cleanup only touches spaces before those listed punctuation marks.
- 1Take each integer ID in the list
- 2Look it up in
int_to_strto recover its text piece - 3Join all the text pieces with single spaces
- 4Remove the space placed before each punctuation mark in the fixed set
Where the Round Trip Changes Spacing
The step most likely to change the text is the join. Because every piece is joined with a space and only spaces before listed punctuation are removed, a space that the original text did not have can remain in the output. The source describes two cases: the space after an opening quotation mark and the space after an apostrophe. Both survive the cleanup in the way described.
"Hi"Original text with the quotation mark directly before Hi
" Hi"Decoded text with a space after the opening quotation mark
In the source passage, the recovered output starts with an opening double quotation mark followed by a space, and the apostrophe in the word It’s has a space after it. The words are all present, and the commas and periods are attached to the words they follow. The mismatch is narrow and comes from a specific step rather than the whole method.
Mistakes Beginners Make
Common mistake Expecting decode to return the exact original string
What it looks likeEncoding “The cat, sat.” and decoding the IDs can return text whose spacing differs from the input near quotation marks and apostrophes.
Why it failsDecode rebuilds spacing with a join and a cleanup step, so it does not promise identical spacing.
Do this insteadCheck the words and tokens separately from the spacing when you compare the round trip.
Common mistake Assuming the vocabulary is case-insensitive
What it looks likeA vocabulary that contains The will not produce an ID for the, because the lookup is a plain dictionary access.
Why it failsThe vocabulary is case-sensitive, so the word must appear in the dictionary exactly as it appears in the input.
Do this insteadMake sure the vocabulary entries match the capitalization used in the input text.
Common mistake Forgetting that empty pieces are removed
What it looks likeTracing the split on “The cat” and including the space piece in the lookup list.
Why it failsEach piece is stripped first, and any piece that ends up empty is removed before the lookup.
Do this insteadStrip each piece and drop empties before looking up IDs.
Common mistake Looking for punctuation to disappear during encode
What it looks likeBelieving the comma in “cat,” is discarded when encoding.
Why it failsThe capture group keeps the boundary characters as pieces, so punctuation becomes its own token if the vocabulary contains it.
Do this insteadExpect punctuation marks to appear as separate pieces and to need their own vocabulary entries.
Common mistake Searching the original vocabulary backwards during decode instead of using the inverse dictionary
Practice Tracing Encode
Using the illustrative vocabulary {The: 0, cat: 1, ',': 2, sat: 3, '.': 4}, trace encode on the string “The sat, cat.” Write the pieces after the split, the pieces after stripping and dropping empties, and the final list of IDs. Then decode your list and state whether the decoded text matches the input exactly.
Hints
- Start by listing every boundary character as its own piece, including the spaces.
- After stripping, the space pieces become empty and must be removed.
- The decoded text joins the words with single spaces and then removes the space before the comma and period.
Key Takeaways
- A tokenizer class implements encode and decode and holds a vocabulary that maps text tokens to integer IDs, plus an inverse dictionary built from it when the class is created.
- Encode splits the text on punctuation, the double-hyphen sequence, and whitespace, keeps the boundary characters as pieces, strips each piece, drops empty pieces, and looks each remaining piece up to produce IDs.
- Encode can only succeed for pieces already in the vocabulary, so the vocabulary is a hard boundary.
- Decode maps IDs through the inverse dictionary, joins the pieces with single spaces, and removes spaces before a fixed set of punctuation marks.
- The join and cleanup steps are where the recovered text can differ from the original in spacing, such as the space after an opening quotation mark or an apostrophe.