Converting a Single Token ID into an Embedding Vector

A language model does not read text directly. Text is first split into tokens, and each token is represented by an integer token ID. Those integers are not yet useful for computation on their own, so the model converts each one into an embedding vector, which is a list of numbers it can work with.

Earlier, you saw one token ID turned into one embedding vector. A language model never sees a single token in isolation, though. It receives a sequence of token IDs, and each one needs its own embedding. This article takes the step you already know and applies it to a whole list of IDs at once.

The running example has four input tokens whose IDs are 2, 3, 5, and 1. The stored IDs are placed in a tensor, which is the data container PyTorch uses for numbers. torch.tensor() creates one from a Python list. The variable input_ids holds the integers that the embedding layer will look up.

Before reading the explanation, commit to an answer. You know that each token ID becomes one embedding vector with three values, and that four token IDs are being passed in together. Ask yourself: how many rows will the output have, and how many values will each row contain?

If your prediction was $4 \times 3$, you reasoned from the structure correctly. If you got a different layout, check which dimension you assumed to be the token count and which you assumed to be the embedding size. The row count comes from the input, and the column count comes from the embedding dimension.

The diagram shows the relationship between inputs and outputs. The first output row always corresponds to the first token ID, because the output keeps the same order as the input. The weight matrix has other rows that are not drawn here, since only the rows selected by these four IDs matter for this call.

The rows of the output are not computed from the token IDs by arithmetic. The embedding layer holds a weight matrix, and each token ID acts as a position to fetch from it. The fetched row is the embedding for that token. The output matrix is simply these fetched rows, placed in the same order as the input IDs.

The lookup is zero-based. Row numbering starts at index 0, so the first row of the weight matrix is row 0. A token ID of 3 therefore selects the row at index 3, which is the fourth row, not the third. Keeping this in mind prevents the most common off-by-one confusion when reading an embedding matrix.

Next, apply the embedding layer to a single token ID. The call passes a tensor that holds only the ID 3. Before running it, notice what is being asked: one token ID goes in, and the layer must return the vector that belongs to that token. The layer is named embedding_layer in this example, and calling it like a function is how it is used.

The returned result is displayed inside double square brackets. The outer pair means the value is a two-dimensional tensor with a single row, and the three numbers inside that row are the components of the embedding vector for token ID 3. Each embedding vector in this example has three components.

The display also includes a label named grad_fn, with the value EmbeddingBackward0. The source does not explain that label in this section, so treat it as metadata that PyTorch attaches to the result. The important part is the three numbers, which are the values stored in the matching row of the weight matrix.

When a lookup result looks wrong, you do not need to guess. Trace three quantities in order: the input token ID, the zero-based row index that the ID selects, and the returned vector. Comparing these three shows the step where your expectation and the actual result diverge.

The check is most useful at the decision point between the ID and the row. If the ID is 3 but you expected the second row, the mismatch comes from counting rows from one instead of zero. If the row index is right but the vector differs from what you expected, the comparison should be made against the matching row of the embedding matrix itself.

The matrix built by the lookup tells the model what each token is, but nothing in it says where that token sits in the text. The same token ID can appear at the start or at the end of a sentence, and the lookup returns the same embedding either way.

For this reason, the embedding matrix holds token identity but not position. The next step adds positional information to these embedding vectors, so that the model can distinguish the same token appearing in different places in a sequence.