Adjusting the Tokenizer for Special Tokens
A special token is added to the vocabulary by hand, because the regular word-building step cannot discover it from the text.
A Vocabulary With a Gap
A tokenizer turns text into integers, and the vocabulary is the lookup table that makes this possible. Each unique token found in the training text receives an integer ID, so the vocabulary is a mapping from token strings to integers. The catch is that this mapping only contains words that actually appeared in the training text. It has no entry for markers the tokenizer itself may need to use.
This article fixes that gap. You will start from a vocabulary that has 1,130 entries, extend its token list with two special tokens, rebuild the mapping, and then check the result. Along the way you will see exactly which integer IDs the new tokens receive and how to confirm they were really added.
Why Special Tokens Are Needed
Special tokens are markers that are not ordinary words. The source names two of them: '<|endoftext|>' and '<|unk|>'. Because they are not part of the regular text, a vocabulary built only from the unique words will never contain them unless you add them deliberately. The tokenizer needs them to exist as entries with their own integer IDs, so it can refer to them when it encodes and decodes text.
A special token is added to the vocabulary by hand, because the regular word-building step cannot discover it from the text.
The Word-Only Starting Point
Before any special tokens are added, the vocabulary is built from the unique tokens in the preprocessed text. The first step sorts those unique tokens and turns them into a list. The third step, enumerate(), then walks through that list and assigns each token the next integer, starting from zero. Because enumerate() counts from zero, the first token gets ID 0 and the last regular token gets the largest ID in the set.
1130set(preprocessed)keeps one copy of each unique tokensorted(list(...))turns those unique tokens into an ordered listenumerate()pairs each token with its position, counting from zero- The starting vocabulary holds 1,130 entries, so its IDs run from 0 to 1129
Extending the Token List
The update uses extend() to attach the two special tokens to the end of the sorted list. The key detail is timing. Because extend() runs after sorting, the special tokens are never sorted in with the ordinary words. They always sit after every regular token, no matter what characters they contain.
1132- Line 1 builds the sorted list of regular unique tokens
- Line 2 appends the two special tokens after every regular token
- Line 3 rebuilds the mapping, so the new tokens get the next integer IDs
- Line 4 checks the size, which should have risen by exactly two
- Line 5 prints the final entries, where the special tokens should appear last
What do you think happens?
The starting vocabulary has 1,130 entries and the code adds exactly two new strings. What size will the final print statement show?
Reveal answer
1132
Two new strings are appended to a list of 1,130 tokens, and each new string becomes one new entry in the mapping. Commit to your own answer before reading this one.
Tracing the Indices After the Update
Trace the indices step by step. The regular tokens still occupy IDs 0 through 1129, exactly as before, because the sorted list is unchanged at its front. The two appended strings sit at positions 1130 and 1131 in the list, and enumerate() gives them those same integers. The size rises from 1,130 to 1,132 as $1130 + 2 = 1132$.
Checking the size and the new IDs
The vocabulary starts with 1,130 entries. After extending the token list with '<|endoftext|>' and '<|unk|>', what size does it have, and which IDs do the special tokens receive?
- 1Start from the old size
The regular tokens occupy IDs 0 through 1129, so 1,130 entries exist before the update.
- 2Count the added strings
extend()adds exactly two strings to the list, one for '<|endoftext|>' and one for '<|unk|>'. - 3Compute the new size
$1130 + 2 = 1132$, so the final vocabulary holds 1,132 entries.
- 4Assign the IDs
The two appended tokens sit at the end of the list, so
enumerate()assigns them 1130 and 1131.
The vocabulary size is 1,132. '<|endoftext|>' has ID 1130 and '<|unk|>' has ID 1131.
Verifying the Update by Printing
A size check tells you how many entries exist, but not which ones. That is why the second check prints the last five items. Because the special tokens were appended last, they should be the final two items in that view. If they are missing, or if they appear earlier in the list, the update did not work as intended.
print(len(vocab))
print(list(vocab.items())[-5:])- 1The first print reports the total number of entries, which should be 1132.
- 2The second print slices the final five pairs. The last two pairs should be the special tokens with IDs 1130 and 1131, and the three before them should be regular tokens with their earlier IDs.
len(vocab)counts the entries, so it confirms the size grew by twolist(vocab.items())[-5:] takes the last five token-ID pairs
1132Where the Special Tokens Sit in the Order
The position of the special tokens follows directly from the order of the operations. Sorting happens before extend(), so the regular words are arranged alphabetically or by their sorted order, and the special tokens are placed after all of them. Their IDs are therefore fixed by how many regular tokens existed before the update.
Mistakes That Break Special Tokens
Three kinds of error are common when special tokens are added. A token can be missing, added twice, or misspelled. Each leaves a different signature when you print the size and the final entries.
Common mistake Forgetting the extend() call
What it looks likeThe code builds the mapping from the sorted list without appending the special tokens.
Why it failsThe size stays at 1,130 and the last entries are regular words, so neither special token exists in the vocabulary.
Do this insteadCall extend() with both special tokens before rebuilding the mapping, then confirm that the size reads 1,132.
Common mistake Adding the same special token twice
What it looks likeThe list is extended with '<|unk|>' in two separate places in the code.
Why it failsA dictionary keeps one entry per key, so the repeated string collapses into a single entry, and the size no longer grows by exactly two.
Do this insteadAppend each special token once, and check that the printed size matches the expected 1,132.
Common mistake Misspelling a special token name
What it looks likeThe string is written as '<|endof_text|>' instead of '<|endoftext|>'.
Why it failsThe size still grows by two, so the size check passes, but the vocabulary lacks the exact marker the tokenizer expects.
Do this insteadPrint the last five entries and compare the spelling of each special token character by character.
Predict the Last Entries
Suppose a colleague extends the token list with '<|unk|>' and '<|endoftext|>' in that order, but a typo leaves a third string, '<|eos|>', also appended. Predict the vocabulary size and the last entry before running the code. Then explain how the last five printed entries would reveal the extra token.
Hints
- Count how many strings are appended in total, then add that number to 1,130.
- The appended strings sit at the end of the list, so the last entries are the place to look.
Key Takeaways
- A vocabulary built only from unique training words has no entries for special markers, so tokens such as '<|endoftext|>' and '<|unk|>' must be added deliberately.
extend()appends the special tokens after the sorted regular tokens, andenumerate()then gives them the next IDs, 1130 and 1131 in this example.- Adding two tokens to a 1,130-entry vocabulary produces a final size of 1,132, which is the first check that the update worked.
- Printing the size and the last five entries confirms that special tokens are present and in the expected positions, and it exposes missing, duplicated, or misspelled tokens.