Implementing a Tokenizer
Tokenization cuts text into individual tokens, and a new text sample is cut in the same way before any numbers are assigned.
Why Text Has to Become Numbers
A language model does its arithmetic on numbers, not on words. Before any text can reach the model, each piece of it has to be given a numeric stand-in. This article follows that translation in both directions: first taking text that is new to the system and expressing it as a sequence of token IDs, then turning numeric output back into readable words.
Two ingredients make the conversion possible. The first is a way of cutting text into smaller units called tokens. The second is a lookup structure, the vocabulary, that assigns each distinct token a unique integer label. Once both are in place, converting new text is mostly a matter of looking things up.
A Sentence the System Has Never Seen Before
Suppose a system has already built a small vocabulary from its training text. Five words are in it: the, cat, sat, on and mat. Each word has been paired with one integer, and no integer is shared by two words. Now a new sample arrives: the cat sat on the mat. The question is how the system should express this sample as numbers using the vocabulary it already has.
What do you think happens?
Using the five-entry vocabulary (the=0, cat=1, sat=2, on=3, mat=4), what sequence of token IDs does the sample 'the cat sat on the mat' produce?
Reveal answer
0, 1, 2, 3, 0, 4
Each word is looked up in the vocabulary in its original order. The word 'the' appears twice, so it receives ID 0 both times, which is why the sequence repeats 0.
Cutting Text into Tokens
Tokenization cuts text into individual tokens, and a new text sample is cut in the same way before any numbers are assigned. A token is not a fixed property of language in general. It is whatever unit the tokenizer produces from the text in front of it.
The examples in this article use a deliberately simple rule: split on spaces, and keep only plain lowercase words. Real tokenizers can be considerably more sophisticated, but the mapping step that follows works the same way regardless of how the cutting was done.
The Vocabulary as a Lookup Table
A vocabulary is a dictionary that associates each token with a unique integer label. It is built from the entire training set, and once it exists it is not tied to that original set. It can be applied to the training set itself and also to new text samples.
In the five-entry example, the token is the key and the integer is the value. The vocabulary stores each token only once, even though a sentence might use the same word several times. That single-entry storage is what lets one lookup table serve every sentence built from its words.
Tracing a New Sample Through the Vocabulary
Converting text to token IDs means looking up each token in the vocabulary and keeping the original order of the tokens. The same cutting rule is applied to the new sample first, and then each resulting token is checked against the vocabulary. The answer is the list of token IDs, and the lookup step is what gives the output its numeric form.
Converting 'the cat sat on the mat'
Using the vocabulary the=0, cat=1, sat=2, on=3, mat=4, produce the token IDs for the sample 'the cat sat on the mat'.
- 1Cut the sample into tokens
Splitting on spaces gives the, cat, sat, on, the, mat. Order is kept.
- 2Look up the first token
'the' is in the vocabulary with ID 0, so the first output is 0.
- 3Look up the next tokens in order
'cat' gives 1, 'sat' gives 2, and 'on' gives 3.
- 4Look up the repeated token
The fifth token is 'the' again. The lookup is the same every time, so it again gives 0.
- 5Look up the last token
'mat' gives 4.
The token ID sequence is 0, 1, 2, 3, 0, 4.
Identical tokens always receive identical IDs, because the lookup is the same every time. This is what allows the numeric sequence to preserve which words recur in the sample.
When a Word Is Missing from the Vocabulary
Running Numbers Back into Text
The conversion so far runs from text toward numbers. But a language model produces numbers as its output, and those numbers have to be turned back into text before a person can read them. The source states this need directly: when the outputs of an LLM are numbers, a way is needed to turn token IDs into text.
An inverse vocabulary maps token IDs back to text tokens. It is the example vocabulary read the other way: given a token ID such as 3, it returns the word on.
Applying this lookup to every ID in an output sequence, in order, reconstructs the text. Running the earlier sentence backwards through the inverse vocabulary takes the sequence 0, 1, 2, 3, 0, 4 back to the words the, cat, sat, on, the, mat. The vocabulary answers 'which number stands for this token?' and the inverse vocabulary answers 'which token does this number stand for?' A system that converts text in both directions needs both.
Mistakes When Converting Text to IDs
Common mistake Assigning a new ID to a repeated word
What it looks likeGiving the second 'the' in 'the cat sat on the mat' a fresh number.
Why it failsThe vocabulary stores each token only once, and the lookup is the same every time, so identical tokens always receive identical IDs.
Do this insteadLook the repeated token up again and reuse its existing ID, here 0.
Common mistake Reordering tokens during conversion
What it looks likeProducing 0, 4, 2, 3, 0, 1 for the sample instead of 0, 1, 2, 3, 0, 4.
Why it failsConverting text to token IDs means keeping the original order of the tokens.
Do this insteadProcess the tokens from left to right and emit one ID per token in the same position.
Common mistake Cutting the new sample differently from the training text
What it looks likeSplitting a new sentence on punctuation when the training text was split only on spaces.
Why it failsA new sample is cut in exactly the same manner, so its pieces can be compared with the pieces the vocabulary already knows.
Do this insteadApply the same cutting rule to new text that was used to build the vocabulary.
Common mistake Treating the inverse vocabulary as a second, separate source of meaning
Why it failsThe inverse vocabulary is simply the example vocabulary read the other way.
Trace a New Sentence Yourself
Using the vocabulary the=0, cat=1, sat=2, on=3, mat=4, produce the token IDs for the sample 'mat sat the'. Then run your answer back through the inverse vocabulary to confirm you recover the original words in order.
Hints
- Split the sample on spaces first, keeping the order: mat, sat, the.
- Look up each token in turn: mat gives 4, sat gives 2, the gives 0.
- The inverse vocabulary maps 4 to mat, 2 to sat and 0 to the, so reading the IDs back in order should restore mat sat the.
Key Takeaways
- Tokenization cuts text into individual tokens, and a new sample is cut in exactly the same way before any numbers are assigned.
- A vocabulary is a dictionary that gives each token a unique integer; converting new text means keeping the original order and looking up each token.
- Identical tokens always receive identical IDs, so the numeric sequence preserves which words recur.
- An inverse vocabulary maps token IDs back to text tokens, which is needed because a model's numeric outputs must be read back as words.
- The source does not specify how a missing word should be handled, so that case must be checked in the specific tokenizer being used.