Backpropagation in Neural Network Layers

Before any model learns anything, each token in a text has already been reduced to an integer. That integer, the token ID, is only a label. It tells the model which entry of its vocabulary the token corresponds to, but it carries no information about meaning. An embedding layer turns that bare label into a vector of numbers that the model can work with.

The question this article answers is how the layer performs that conversion. The short answer is that it does not compute anything complicated. It looks up a row. The rest of the article explains what that means, why the lookup gives exactly the vector you expect, and how it connects to a different way of writing the same operation.

The embedding layer holds a weight matrix, and each row of that matrix stores the vector for one token. The token ID is used as a position. The layer moves to that position in the weight matrix and reads the row found there. Rows at other positions are not touched for this lookup.

The diagram shows the same idea in a different form. The five rows are numbered from zero, so the row with position 3 is the fourth row. The values inside the rows are illustrative numbers chosen for this lesson; the point is that only the highlighted row is read.

Token IDs are zero-based row indices. The first row has index 0, so token ID 0 retrieves the first row, and token ID 3 retrieves the fourth row. Counting from one is the most common source of confusion when checking an embedding by hand.

Readers who already know one-hot encoding may recognize the lookup as a shortcut. One-hot encoding represents a token ID as a vector whose length equals the vocabulary size, containing a single 1 at the position of the token and 0 everywhere else. That vector is then multiplied by a weight matrix in a fully connected layer.

Take a vocabulary of five tokens. The one-hot vector for token ID 3 has a 1 in position 3, counting from zero, and zeros in the other four positions. Multiplying it by the weight matrix produces the row at index 3, because every other term in each sum is multiplied by zero. The result is the same row the embedding layer would have returned directly.

In this formula, x is the one-hot vector for token ID 3, W is the weight matrix, and W_{3,:} is its row at index 3. The one-hot multiplication and the direct lookup therefore give identical vectors.

Verification is simple once the indexing is clear. You take the token ID, find the row at that zero-based position in the weight matrix, and compare the stored numbers with what the layer returned. The returned vector is not approximated, so the match should be exact.

When an embedding result looks wrong, follow the path from token ID to returned vector and check each step. The important consequence from the source is that the returned vector is identical to the stored row, so a mismatch points to the index used for the lookup rather than to some hidden calculation.

The divergence almost always occurs at the index step. If the vector returned for token ID 3 matches the third row instead of the fourth, the index used for the lookup was off by one. The matrix values and the returned copy are not the place to look first. Check which position the lookup actually used, then count rows from zero to confirm the expected row.