Converting Tokens to IDs

A vocabulary maps each unique token to a unique integer called a token ID, and it is built from the tokens in the training text.

Programming

Why Text Must Become Numbers

Text models do not read words. Before a model can process a passage, every token has to become a number. The first step in that conversion is a vocabulary, a lookup table that assigns each distinct token a fixed integer. A language model does its arithmetic on numbers, not on words, so each piece of text needs a numeric stand-in before it can reach the model.

This article follows the translation in both directions. First you will see how a vocabulary is built from a small list of tokens, then how new text is converted into a sequence of token IDs using that vocabulary, and finally how numeric output is turned back into readable words. The fastest way to understand a vocabulary is to watch what the code does to a real list of tokens and then reason about each stage.

Predicting a Vocabulary for a Small Sentence

Consider the tokenized sentence below, where each space-separated word is one token. Tokenization is the step that produces this list. Before you read the answer, work it out yourself: how many unique tokens does this sentence contain, and which ID would the word 'the' receive if the vocabulary is built by sorting the unique tokens alphabetically and numbering them from zero by position?

python

sentence = "the cat sat on the mat"
preprocessed = sentence.split()
print(preprocessed)

sentence = "the cat sat on the mat"
preprocessed = sentence.split()
print(preprocessed)

Output (expected)

['the', 'cat', 'sat', 'on', 'the', 'mat']

Notes on the code

split() cuts the string on spaces, so each space-separated word becomes one token
The word 'the' appears twice in the list, so the list has six entries but only five distinct tokens

What do you think happens?

How many unique tokens are in this sentence, and what ID does 'the' receive?

4 unique tokens; 'the' gets ID 0
5 unique tokens; 'the' gets ID 4
6 unique tokens; 'the' gets ID 4

Reveal answer

Answer:

5 unique tokens; 'the' gets ID 4

The six-item list contains 'the' twice, so only five distinct tokens remain: the, cat, sat, on, mat. Sorted alphabetically they become cat, mat, on, sat, the. IDs are assigned by position in that sorted list, so 'the' sits at position 4 and receives ID 4.

Splitting Text into Tokens

Two ingredients make the conversion possible. The first is a way of cutting text into smaller units called tokens. The second is a lookup structure, the vocabulary, that assigns each distinct token a unique integer label. Tokenization cuts text into individual tokens, and the cutting rule is what decides what a token is.

A token is therefore not a fixed property of language in general. It is whatever unit the tokenizer produces from the text in front of it. The examples in this article use a deliberately simple rule: split on spaces, and keep only plain lowercase words. Real tokenizers can be considerably more sophisticated, but the mapping step that follows works the same way regardless of how the cutting was done.

The Four Stages of Vocabulary Building

Vocabulary creation is a sequence of transformations, and each stage takes the output of the previous one. The construction runs in four stages: the tokens as tokenized, the set of unique tokens after duplicates are removed, the sorted list, and finally the mapping from each token to its ID. A mistake at any point propagates forward, so it helps to check each stage separately.

What does the list look like after each stage, and at which stage would the output first differ from what I expected?

The vocabulary size is the length of the deduplicated, sorted list. It counts unique tokens rather than total occurrences, which is why the six-word sentence above has a vocabulary size of 5, not 6. The word 'the' contributes to the count once, however many times it appears in the text.

From Sorted List to Token IDs

The first two lines of the vocabulary code reduce the token list to its distinct members and put them in order. The next step turns the sorted list into the mapping. Its central line is a dictionary comprehension. Read it from the inside out: enumerate(all_words) pairs each token with its position, and the comprehension stores each pair with the token as the key and the integer as the value.

Code walkthrough

preprocessed = sentence.split()
unique_words = set(preprocessed)
all_words = sorted(unique_words)
vocab_size = len(all_words)
vocab = {word: i for i, word in enumerate(all_words)}
print(vocab_size)
print(vocab)

1
Line 1 splits the sentence into its six tokens, one of which ('the') appears twice.
2
Line 2 removes duplicates with set, leaving five distinct tokens in no guaranteed order.
3
Line 3 sorts those five tokens alphabetically, giving cat, mat, on, sat, the.
4
Line 4 counts the sorted list, so the vocabulary size is 5.
5
Line 5 pairs each token with its position using enumerate, then stores token as key and position as value. The key is the token and the value is the integer, so the order of these two matters.
6
Line 6 and 7 print the size and the finished mapping.

set(preprocessed) keeps one copy of each distinct token and discards the repeated 'the'
sorted(unique_words) puts the five tokens in alphabetical order
len(all_words) counts unique tokens, giving the vocabulary size

Output

5
{'cat': 0, 'mat': 1, 'on': 2, 'sat': 3, 'the': 4}

The IDs come from position in the sorted list, not from meaning or frequency. 'cat' is first alphabetically, so it receives ID 0. 'the' appears most often in the sentence but is last in sorted order, so it receives the highest ID, 4. Changing the sort order would change every ID, even though the words and their relationships stayed the same.

When the sorted unique tokens are numbered, which position in the list becomes which ID, and why does changing the sort order change every ID?

The same procedure applies to a full literary text. The source applies it to Edith Wharton's short story after tokenizing it and storing the tokens in a variable called preprocessed. The vocabulary size for that text is 1,130, and the first 51 entries are printed for inspection. The procedure is identical to the small example; only the scale changes. A real vocabulary is simply a longer sorted list of unique tokens.

Where a Result Can Diverge

When a printed vocabulary looks wrong, check the four stages in order: tokens, unique tokens, sorted tokens, IDs. Compare each stage with what you expected. The first stage whose output differs from your expectation is the one to revisit. For a result to be wrong, one of the four stages must have produced something other than what you expected, and tracing the stages is how you find it.

Common mistake
Counting repeated tokens toward the vocabulary size
What it looks like
Counting six words in 'the cat sat on the mat' and reporting a vocabulary size of 6.
Why it fails
The vocabulary size is the length of the deduplicated list, which counts unique tokens rather than total occurrences.
Do this instead
Count only distinct tokens. The size here is 5.
Common mistake
Reversing the key and value in the dictionary comprehension
What it looks like
Writing {i: word for i, word in enumerate(all_words)}, which maps IDs to words.
Why it fails
The comprehension must store the token as the key so that a token lookup returns its ID. Reversing them breaks the lookup.
Do this instead
Use {word: i for i, word in enumerate(all_words)} so each token is the key and its integer is the value.
Common mistake
Expecting IDs to follow frequency or meaning
What it looks like
Assuming 'the', the most frequent word, should get ID 0.
Why it fails
IDs are assigned by position in the alphabetically sorted list, not by meaning or how often a word occurs.
Do this instead
Sort first, then number by position. 'the' is last alphabetically, so it gets ID 4.
Common mistake
Skipping the sort and assuming set() returns tokens in the original order

Tracing a New Sample Through the Vocabulary

Once a vocabulary exists, it is not tied to the text it was built from. It can be applied to the training text and also to new samples. Converting a new sample has two steps. First, the sample is tokenized with the same cutting rule. Second, each resulting token is looked up in the vocabulary, and the IDs are collected in the original order of the tokens.

When a new sentence contains tokens, how does each token move through the vocabulary lookup to produce its token ID sequence?

Worked example

Converting 'the mat sat' with the vocabulary from the sentence above

Using the vocabulary cat=0, mat=1, on=2, sat=3, the=4, convert the new sample 'the mat sat' into token IDs.

1
Tokenize the sample
Split on spaces and keep plain lowercase words, giving the tokens the, mat, sat in that order.
2
Look up 'the'
'the' is a key in the vocabulary with value 4, so its first ID is 4.
3
Look up 'mat'
'mat' maps to 1, so the second ID is 1.
4
Look up 'sat'
'sat' maps to 3, so the third ID is 3. The order of the IDs matches the order of the tokens.

Result

The token ID sequence is 4, 1, 3.

Identical tokens always receive identical IDs, because the lookup is the same every time. If 'the' appears three times in a sample, the ID 4 appears three times in the output sequence, which is what allows the numeric sequence to preserve which words recur.

Reading Numeric Output Back into Text

The conversion so far runs from text toward numbers. A language model produces numbers as its output, and those numbers must be turned back into text before a person can read them. An inverse vocabulary maps token IDs back to text tokens. It is simply the vocabulary read the other way: given an ID, it returns the token.

Applying the inverse lookup to every ID in an output sequence, in order, reconstructs the text. In the source's illustration, the vocabulary assigns the=0, cat=1, sat=2, on=3, and mat=4. Running the sequence 0, 1, 2, 3, 0, 4 backwards through the inverse vocabulary gives the words the, cat, sat, on, the, mat. Note that this hand-made illustration does not follow alphabetical order; the IDs were chosen by hand, while the code above derives IDs from sorted position. Both use the same lookup mechanism in opposite directions.

Structure	Question it answers	Direction	Example
Vocabulary	Which number stands for this token?	Text to number	sat becomes 3
Inverse vocabulary	Which token does this number stand for?	Number to text	3 becomes sat

The vocabulary and the inverse vocabulary are mirror images, each answering a different question.

A system that converts text in both directions needs both structures. The vocabulary is needed before text can reach the model, and the inverse vocabulary is needed after the model produces numbers. Without the inverse step, a model's output would remain a list of integers that a reader cannot interpret.

Practice Tracing a Sample

EASY

Using the vocabulary cat=0, mat=1, on=2, sat=3, the=4, convert the sample 'the on the sat' into its token IDs. Then state how many distinct tokens the sample contains and whether the repeated 'the' receives the same ID both times.

Hints

Tokenize first by splitting on spaces, keeping the order of the words.
Look up each token in the vocabulary; repeated tokens return the same ID every time.
Distinct tokens are counted once each, so 'the' contributes to the distinct count only once.

Your attempt (not saved)

Key Takeaways

A vocabulary maps each unique token to a unique integer called a token ID, and it is built from the tokens in the training text.
Vocabulary building runs in four stages: tokens, unique tokens (via set), sorted tokens, and IDs assigned by position; the vocabulary size counts unique tokens, not total occurrences.
In the dictionary comprehension the token is the key and the integer is the value; reversing them breaks the lookup.
New text is tokenized with the same rule, then each token is looked up to produce its ID sequence in original order.
An inverse vocabulary maps token IDs back to text tokens, which is needed to turn a language model's numeric outputs into readable text.

Related concepts

More concepts starting with C