Token IDs and the Vocabulary

Before a model learns anything, every token in a text has already been reduced to an integer. That integer is the token ID. It tells the model which entry of its vocabulary the token corresponds to, but on its own it carries no information about meaning. A token ID is only a label, the way a seat number tells you where a passenger sits without describing the passenger.

A neural network cannot do much with a bare label, so an embedding layer converts each token ID into a vector of numbers. The question this article answers is how that conversion happens. The short answer is that the layer does not perform a complicated calculation. It looks up a row. The rest of the article explains what that means, why the lookup returns exactly the vector you expect, and how it connects to a different way of writing the same operation.

Consider a small embedding matrix with five rows, one for each vocabulary entry. The rows are numbered 0 through 4. Suppose the token ID you pass in is 3. Before reading on, decide which row you think comes back, and whether the layer computes anything new to produce it.

The weight matrix of an embedding layer has one row per vocabulary entry and one column per embedding dimension. Each token in the vocabulary therefore owns exactly one row. When the layer receives a token ID, it does not inspect the other rows for this lookup. It reads only the row whose position matches the ID.

The diagram shows a five-token vocabulary in which each row is labeled with its token and its three embedding values. Token ID 3 is the token 'on', and it selects the highlighted row at position 3. The values are illustrative numbers chosen for this example, not values from a trained model.

The verification step is the most useful habit to build. Because the returned vector is a copy of a stored row, you can check any embedding output against the matrix. If the values differ, the problem lies in the index that was used, not in some hidden calculation.

Readers who know one-hot encoding may recognize the lookup as a shortcut. One-hot encoding represents a token ID as a vector whose length equals the vocabulary size. That vector holds a single 1 at the position of the token and 0 everywhere else. The one-hot vector is then multiplied by a weight matrix in a fully connected layer.

For the five-token vocabulary, the one-hot vector for token ID 3 has a 1 in position 3 and zeros in the other four positions. Written as a row vector, it is multiplied by the matrix as follows.

Here $\mathbf{x}$ is the one-hot vector, $W$ is the weight matrix, and $W_{3,:}$ is the row at index 3. Each entry of the product is a sum of terms, and every term except one is multiplied by zero. Only the term that multiplies the 1 survives, so the product reproduces the row at index 3. This is the same row the embedding layer would have returned directly.

The two paths produce the same output. The one-hot path does extra work: it builds a vector of vocabulary length, multiplies it against every row, and adds up terms that are mostly zero. The lookup path skips all of that and reads one row. This is why the source describes the embedding layer as the more efficient implementation of the same operation.

When an embedding output does not match what you expected, follow the path from token string to token ID to matrix row to returned vector. The mismatch appears at exactly one step, and checking each step in order will find it. The returned vector is a copy of a stored row, so a wrong output almost always means the wrong row was read.

This procedure works because each step has a checkable output. You can confirm the token ID against the vocabulary, the position against the zero-based count, and the returned values against the matrix row. The step where the check fails is the step that diverges.