Creating a Vocabulary from a Training Set

A language model does its arithmetic on numbers, not on words. Before any passage of text can reach the model, each piece of it has to be given a numeric stand-in. This article follows that translation in both directions: first taking text that is new to the system and expressing it as a sequence of numbers, then turning numeric output back into readable words, because a model whose output cannot be read is of little use.

Two ingredients make the conversion possible. The first is a way of cutting text into smaller units called tokens. The second is a lookup structure, the vocabulary, that assigns each distinct token a unique integer label. Once both are in place, converting new text is mostly a matter of looking things up.

Tokenization is the step that cuts text into individual tokens. A token is not a fixed property of language in general. It is whatever unit the tokenizer produces from the text in front of it. This article uses a deliberately simple rule: split on spaces, and keep only plain lowercase words. Real tokenizers can be considerably more sophisticated, but the mapping step that follows works the same way regardless of the cutting rule.

Read the pipeline from left to right. The raw tokens list may contain many repeated entries. The first transformation collapses those repeats so that each distinct token appears once. The second sorts what remains. The third step, assigning IDs, depends on that order, which is why the sort is not optional decoration. Each stage takes the output of the previous one, so a mistake at any point carries forward to every later stage.

The construction runs in four stages: tokenize, remove duplicates with set, sort alphabetically, and assign IDs by position. The first two stages produce the list of unique tokens. The third stage arranges that list in a fixed order, and the fourth stage numbers its entries. The code below follows those steps for a sentence whose tokens you can count by hand.

Notice what the mapping does and does not depend on. The IDs are not chosen by meaning or by how often a word appears; 'the' appears twice in the sentence but still receives a single ID, 3, because the vocabulary stores each token only once. The IDs simply follow alphabetical position in the sorted list. Because the lookup is the same every time, identical tokens always receive identical IDs.

The source applies the same procedure to a full literary text, Edith Wharton's short story. After tokenizing it and storing the tokens in a variable called preprocessed, the code determines that the vocabulary size for that text is 1,130, and the vocabulary's first 51 entries are printed for inspection. The procedure is identical to the small example; only the scale changes. A real vocabulary is simply a longer sorted list of unique tokens.

Tokenization is also the first step when a new text sample arrives. The sample is cut into tokens in exactly the same manner as the training set, so its pieces can be compared with the pieces the vocabulary already knows about. Converting the sample means looking up each token in the vocabulary and keeping the original order of the tokens. The output is a list of token IDs in the same sequence as the words.

This example depends on the vocabulary already containing every word in the new sample. The section this article draws on does not specify how a conversion should behave when a word is missing from the vocabulary, so no answer is assumed here. In plain Python, a dictionary lookup for a key that does not exist raises an error, which is one reason the handling of unseen tokens is a design decision that has to be made deliberately.

The conversion so far runs from text toward numbers. But a language model produces numbers as its output, and those numbers have to be turned back into text before a person can read them. The source states this need directly: when the outputs of an LLM are numbers, a way is needed to turn token IDs into text. The inverse vocabulary provides that way.

The inverse vocabulary is the example vocabulary read the other way. Given a token ID such as 3, it returns the word on in the source's example. Applying this lookup to every ID in an output sequence, in order, reconstructs the text. In the source's example, the sequence 0, 1, 2, 3, 0, 4 is run back through the inverse vocabulary to recover the words the, cat, sat, on, the, mat. That example uses a hand-written vocabulary whose IDs follow the table given in the source rather than the alphabetical order produced by the sorting code, so the two should not be confused.