Building a Vocabulary from Unique Tokens
The vocabulary is a mapping from token strings to integers. Anything that is not in that mapping cannot be assigned an ID, which is why special markers must be added deliberately.
A Vocabulary Without Its Markers
A tokenizer needs a vocabulary, which is a mapping from token strings to integer IDs. In the pipeline this lesson continues, the vocabulary was built from the unique words found in the preprocessed training text. Every unique token received an integer ID, so the vocabulary contains only words that actually appeared in that text. It has no entry for any marker that the tokenizer itself may need to use.
This article adds two such markers to the existing vocabulary: '<|endoftext|>' and '<|unk|>'. Before you read on, consider the setup. The previous vocabulary had 1,130 entries, and the update adds exactly two new strings to the list of tokens.
What do you think happens?
Before reading on: the vocabulary starts with 1,130 entries and two special token strings are added. What size will the final check print?
Reveal answer
1,132
The two special tokens are appended to the existing list of 1,130 tokens, so the vocabulary grows by exactly two entries to 1,132. The source reports that the vocabulary size rose from 1,130 to 1,132.
Why the Vocabulary Needs Special Markers
A special token is a marker string, such as '<|endoftext|>' or '<|unk|>', that is added to the token list after the regular unique words so that the vocabulary can represent something other than ordinary words.
The reason for adding them is that the vocabulary built from the training text has no entry for markers the tokenizer may need. Because the vocabulary is only a mapping from strings to integers, a marker that is not in the mapping has no integer ID at all. Adding the markers explicitly gives them IDs of their own, so the tokenizer can refer to them like any other token.
The vocabulary is a mapping from token strings to integers. Anything that is not in that mapping cannot be assigned an ID, which is why special markers must be added deliberately.
Tracing the Three Moves
The update follows three moves, and the code you run follows the same sequence. First, take the existing list of unique tokens. Second, append the special tokens to that list. Third, rebuild the token-to-integer mapping. A fourth check then confirms that the new tokens made it in.
- 1Sort the unique tokens and turn them into a list. The regular words now occupy the list in sorted order.
- 2Call
extend()to attach '<|endoftext|>' and '<|unk|>' to the end of that list. Becauseextend()runs after sorting, the special tokens are not sorted in with the ordinary words. - 3Build the dictionary with
enumerate(). Enumerate counts from zero, so each token receives the next integer in turn, and the last two list items receive the last two integer IDs. - 4Check the size of the vocabulary. It should now be 1,132 entries rather than 1,130.
- 5Print the last few entries. The special tokens should appear as the final two items.
The diagram shows the index assignment that results from the three moves. The regular words keep the IDs they had before the update, since they sit at the front of the list. The two special tokens follow them: '<|endoftext|>' receives index 1130 and '<|unk|>' receives index 1131. Together they account for the rise from 1,130 entries to 1,132.
Why Appending Places Specials Last
The order of operations matters. The unique tokens are sorted first, and only then is extend() called. Sorting therefore applies only to the regular words, and the special tokens are attached afterward. They always sit after every regular token. The enumerate() step then counts from zero through the whole list, so the special tokens, which come last in the list, are assigned the last integer IDs.
Reading the Size and the Final Entries
A size check tells you how many entries exist, but not which ones. For that reason the update is verified in two ways: by the total size and by the final entries. The second check prints the last five items, and because the special tokens were appended last, they should be the final two items in that view. If they are missing, or appear earlier in the list, the update did not work as intended.
Confirming the special tokens were added
The vocabulary started with 1,130 entries. After the update, check the size and the final entries to confirm that '<|endoftext|>' and '<|unk|>' were incorporated.
- 1Start from the old size
The regular unique tokens occupy indices 0 through 1129, giving 1,130 entries before the update.
- 2Append two strings
extend()adds exactly two new strings, so the list grows by two. - 3Check the size
The printed size should be 1,132, the value the source reports after the update.
- 4Check the final entries
The last two positions should hold '<|endoftext|>' at index 1130 and '<|unk|>' at index 1131.
- 5Check the earlier entries
The entries before them are the regular tokens, which should keep their earlier indices.
The size rose from 1,130 to 1,132, and the special tokens hold indices 1130 and 1131. This is the evidence that the special tokens were successfully incorporated.
| Check | What it confirms | Expected result |
|---|---|---|
| Vocabulary size | The update added entries | 1,132 after the update, up from 1,130 |
| Last five entries | The special tokens sit at the end | The final two items are '<|endoftext|>' and '<|unk|>' |
| Index of '<|endoftext|>' | The special token has its own ID | 1130 |
| Index of '<|unk|>' | The second special token has its own ID | 1131 |
Checks that verify the special tokens were incorporated
Mistakes That Lose, Repeat, or Rename Special Tokens
Common mistake Building the mapping before appending the special tokens
What it looks likeThe dictionary is created from the regular words, and extend() is called afterward on the list without rebuilding the mapping.
Why it failsThe mapping is what gives each token an integer ID. A mapping built before the append has no entries for the special tokens, so they are effectively missing from the vocabulary.
Do this insteadRebuild the token-to-integer mapping after extend() so that the new tokens receive IDs.
Common mistake Expecting the special tokens to be sorted in with the regular words
What it looks likeThe special tokens are placed in the list after sorting, but the learner expects them to appear at alphabetical positions among the words.
Why it failsextend()runs after sorting, so the special tokens always sit after every regular token and receive the last IDs.Do this insteadLook for the special tokens at the end of the vocabulary, at indices 1130 and 1131.
Common mistake Appending a special token twice
What it looks likeThe same marker is passed to extend() twice, or it is appended once more on a later run of the same code.
Why it failsEach appended string adds an entry. The size check will show more than the expected two new entries, and the marker will appear more than once.
Do this insteadConfirm that the size grew by exactly two and that each special token appears once in the final entries.
Common mistake Misspelling a special token
What it looks likeThe marker is written as '<|unk>' or '<unk|>' instead of '<|unk|>'.
Why it failsThe vocabulary is keyed by exact strings. A misspelled marker is a different string, so the intended special token is not found by a lookup, even though an entry with the wrong name is added.
Do this insteadCompare the printed final entries character by character against '<|endoftext|>' and '<|unk|>'.
Common mistake Checking only the size and not the final entries, which cannot show which tokens were added
| Situation | Vocabulary size | Final two entries | Diagnosis |
|---|---|---|---|
| Correct update | 1,132 | '<|endoftext|>' at 1130, '<|unk|>' at 1131 | Both special tokens were incorporated |
| One special token omitted | 1,131 | Only one marker at the end | A token was not appended |
| Token appended twice | More than 1,132 | A marker repeated at the end | A duplicate was added |
| Token misspelled | 1,132 | An entry such as '<|unk>' at the end | The intended marker is missing under its correct name |
How correct and faulty special-token vocabularies differ
Checking Your Own Vocabulary
A vocabulary has 1,130 regular unique tokens. You append the two special tokens '<|endoftext|>' and '<|unk|>' to the list and then rebuild the mapping with enumerate(). Predict the index of '<|unk|>' and the size you should see. Then describe the single printed check that would reveal a misspelled special token.
Hints
- Regular tokens use indices 0 through 1129, so the first appended token takes the next index.
- The size check counts entries, but the final-entries check shows the names.
- Compare the last few printed entries against the exact spelling of each marker.
What the Vocabulary Now Holds
- 1The vocabulary built from the training text has no entries for markers the tokenizer may need, so special tokens such as '<|endoftext|>' and '<|unk|>' are appended explicitly.
- 2The update runs in three moves: sort the regular unique tokens, append the special tokens with
extend(), and rebuild the mapping withenumerate(). - 3The special tokens receive indices 1130 and 1131, and the vocabulary size rises from 1,130 to 1,132.
- 4Verify the update by checking the size and printing the final entries, which should end with the two special tokens.
- 5Common errors are omitting a token, adding it twice, misspelling it, or building the mapping before the append.
Key Takeaways
- Special tokens such as '<|endoftext|>' and '<|unk|>' are added because the training-text vocabulary has no entries for markers the tokenizer may need.
- Calling
extend()after sorting places the special tokens after every regular word, so they receive the last integer IDs, 1130 and 1131. - The vocabulary size rises from 1,130 to 1,132, which is the first check that the update worked.
- Printing the final entries confirms that the special tokens are present in the expected positions.
- Omitted, duplicated, or misspelled special tokens each show up as a distinct mismatch in the size or final entries.