Building a Basic Tokenizer

A tokenizer that works on one sentence has not yet been proven. The real test is whether it still behaves correctly on a much longer text. In this article you apply a basic tokenizer to an entire short story by Edith Wharton and check three things: how many tokens come out, whether the first tokens look right, and how those tokens are then turned into integers.

Before you run anything, notice what your guess is based on. Most people estimate a count by counting words and punctuation marks. That is close to what the tokenizer does, but the gap between a rough guess and the exact count is exactly the kind of detail a step-by-step trace can settle for you. For this story the tokenizer produces 4690 tokens once blank and whitespace-only items are removed.

Run the code that accompanies this section first and watch what it does. You do not need to understand every piece yet. The sections that follow send you back to the trace whenever you need to check one specific stage.

The core of the tokenizer is a single call to the re.split function from Python's re module. You give it a pattern and a string, and it cuts the string wherever the pattern matches. Nothing else in the tokenizer decides where a boundary falls. Everything that follows is a consequence of that one call.

The pattern in this tokenizer matches three kinds of boundary. The first is a character class listing commas, periods, colons, semicolons, question marks, underscores, exclamation marks, parentheses, and quotation marks. The second is the double dash. The third is any whitespace character. Each time the pattern matches one of these, the string is cut at that position.

Take the short text Hello, world. It is small enough to trace by hand, and it has the same structure as the full story: words, punctuation, and spaces. Follow the three stages in order: split, keep the pieces, then remove the empty ones.

Read the pipeline from top to bottom. The first stage decides where the boundaries fall, and the words and punctuation marks come out of that stage in the order they appeared in the text. The second stage only removes items that contain nothing visible. It never reorders or merges anything. That is why the token list keeps the original sequence of the text.

Notice that the punctuation survives in the output of the trace above. That result depends on a detail of the pattern: the parentheses. When a pattern contains a capturing group, re.split returns the matched text along with the pieces between matches. Without the group, the matched characters are used only as cut points and are thrown away.

The left column shows what you get when the pattern has no capturing group. The commas and periods are gone, and the list holds only the words and the empty strings around the cut points. The right column shows the same split with the parentheses in place. The matched punctuation now appears as its own item, sitting between the words it separated.

Because the split happens at every whitespace character, the raw result also contains empty strings and strings that hold only spaces. Those are not tokens. A second pass removes them, keeping only items that still contain something after stripping spaces. This is why the story's count of 4690 tokens is the number of meaningful items, not the number of items the split first produced.

In the diagram, the seven raw items become four tokens. The three dropped items are the two empty strings and the single space. The visible items, the word Hello, the comma, the word world, and the period, are untouched and keep their original order. Counting before this filter would overstate the number of tokens by exactly the number of blank items.

A list of tokens is still a list of Python strings. The next step converts those tokens from strings into an integer representation, producing token IDs. The course describes this conversion as an intermediate step that sits between the tokens and the embedding vectors that come after it. Each unique token is mapped to a single integer, so the token list becomes a list of numbers.

The diagram shows one possible assignment for the four tokens of Hello, world. Each token appears exactly once in the mapping, and it receives one integer. The ID values here were chosen for illustration; the lookup itself works by checking each token in the token list against the mapping and writing down its integer. The result is a list of IDs that stands in for the list of tokens.

Keep the two stages separate in your head. Tokenization decides where the boundaries fall. Token ID conversion decides how each resulting token is represented as a number. The ID step depends on the token list being right, so a boundary error in the first stage carries straight into the IDs. That is why you verify the token list before you trust any IDs built from it. The embedding vectors come after the IDs, so they are built from numbers that already reflect the tokenizer's choices.

When the tokenizer's output looks wrong, the fastest way to find the problem is to check each intermediate result in order rather than rereading the whole pipeline. A quick visual check of the first thirty tokens reveals whether the boundaries fall in the right places. Look at the order of the items, whether any word is cut in half, and whether any punctuation mark is stuck to a word.

The first stage at which the actual output differs from your expected output is the place to look. If the punctuation is missing, the problem is in the capturing group or the character class. If a word has been split in two, a boundary pattern is matching too eagerly. If the count is too high, the filter may not be removing whitespace-only items. Each of these symptoms points to a single stage, which is why checking the pipeline in order is faster than guessing.