Token IDs

A language model does its arithmetic on numbers, not on words. Before any text can reach the model, each piece of it has to be given a numeric stand-in. Suppose you have already split a sentence into tokens and assigned each token an integer. You now have a list of numbers that stands in for the sentence. The question this article answers is what happens next: what does the model receive as its input once those integers exist?

The short answer is that the integers are converted into embedding vectors, and those vectors are what the LLM works with during training. Keep this sequence in mind as we go: text comes first, then tokens, then token IDs, and finally embedding vectors. Each stage changes the form of the data while keeping track of the same content. The last stage is the one that gives the model numeric input it can adjust while it learns.

The preparation path has three transitions. The first tokenizes the text, cutting it into individual tokens. The second converts each token into a token ID, which is a single integer that identifies that token. The third converts each token ID into an embedding vector, which is the representation the model receives as input. The path runs from input text to tokenized text, then to token IDs, then to embedding vectors.

Each stage keeps the same sentence in a different form. Nothing in the content is lost between stages; the data is simply expressed in a more useful shape. The left side of this pipeline is the preparation the article describes. Once the embedding vectors exist, the prepared input moves on into the model itself, which is the only part of the overall path that involves the model.

Two ingredients make the conversion possible. The first is a way of cutting text into smaller units called tokens. The second is a lookup structure, the vocabulary, that assigns each distinct token a unique integer label. Once both are in place, converting new text is mostly a matter of looking things up.

Tokenization is the first step whenever a new text sample arrives. The sample is cut into tokens in exactly the same manner that was used for the training set, so that its pieces can be compared with the pieces the vocabulary already knows about. A token is therefore not a fixed property of language in general. It is whatever unit the tokenizer produces from the text in front of it.

The examples in this article use a deliberately simple rule: split on spaces, and keep only plain lowercase words. Real tokenizers can be considerably more sophisticated, but the mapping step that follows works the same way regardless of how the tokens were cut. Only after the cutting is finished does any number get assigned.

Once the vocabulary exists, it is not tied to the original training set. It can be applied to the training set itself and also to new text samples, which is what makes it useful for the conversion task. Each word is paired with one integer, and no integer is shared by two words. Reading a vocabulary from left to right, the token is the key and the integer is the value.

Notice that the vocabulary stores each token only once, even though a sentence might use the same word several times. The word 'the' appears twice in the sentence used below, but it has one entry and one ID in the vocabulary.

The worked example shows a property worth remembering: identical tokens always receive identical IDs, because the lookup is the same every time. This is what allows the numeric sequence to preserve which words recur. The second 'the' in the sentence produces the same 0 as the first, so a reader of the ID sequence can see the repetition.

The last transition is the least obvious. A token ID is a single integer that identifies a token. An embedding vector is a representation of that token in a form the model can use for learning. The conversion keeps the identity of each token while changing its form, so each ID in the sequence is paired with its own embedding vector. One token ID leads to one embedding vector.

The diagram is a pairing, not a calculation. It tells you which vector belongs to which ID. The internal arithmetic of this lookup is not described in the material, so the idea to hold onto is the correspondence itself. One way to picture it, offered here as a teaching aid rather than as a claim about the implementation, is a table with one row per token ID, where the ID tells you which row to read.

The embedding vectors are not handed to the model already meaningful. The weights that produce them start with random values. These weights are then optimized during LLM training, so the values that begin random are gradually adjusted as the model learns from data.

This is why the last preparation stage matters so much. The integer IDs are fixed labels, but the embedding vectors are the part the model can change. The conversion gives the model a numeric input whose values it can adjust during learning, which is the reason the vectors, and not the IDs, are what the LLM works with during training.

The conversion so far runs in one direction, from text toward numbers. A language model produces numbers as its output, and those numbers have to be turned back into text before a person can read them. A model that cannot be read back into text is of little use, so the reverse journey is part of the same system.

Applying this lookup to every ID in an output sequence, in order, reconstructs the text. Running the earlier sentence backwards takes the sequence 0, 1, 2, 3, 0, 4 back to the words the, cat, sat, on, the, mat. The two structures are mirror images, and each answers a different question. The vocabulary answers 'which number stands for this token?' The inverse vocabulary answers 'which token does this number stand for?' A system that converts text in both directions needs both.