Embedding Layers and Weight Matrices

Imagine a short input that contains the same word twice, once near the beginning and once near the end. As a human reader, you see two copies of the same word and know they are the same word. The question for this article is whether a large language model treats the two copies the same way. The answer depends on what the first stage of the model, the embedding layer, does with each token.

Before reading on, make a guess. When the embedding layer receives two copies of one word, does it return the same vector both times, or two different vectors? Keep your guess in mind, because the rest of this article tests it.

An embedding layer holds a weight matrix. Each row of that matrix is a vector, and each token ID selects exactly one row. When the layer receives a token ID, it does not perform any complicated computation. It retrieves the vector stored in the row that corresponds to that ID. A useful way to picture this is a table in which the token ID is the address of a row. The output of the layer is simply the row that was retrieved.

The lookup is the whole job of the layer. In the illustrated matrix, rows are numbered from the top, starting at zero. Token ID 5 retrieves the sixth row, whose entries are -2.8400, -0.7849, and -1.4096. The same procedure applies to every other ID: each one fetches exactly the row that carries its number, and nothing else in the sequence influences that retrieval.

Because the lookup depends only on the token ID, the layer produces the same vector for a given ID wherever that ID appears. Token ID 5 yields the same embedding vector whether it sits in the first position or the fourth. The embedding layer has no input for position, so position cannot influence its output. This is the key to predicting what happens with repeated words.

Consider a sentence in which fox appears at positions one and four. Both occurrences carry token ID 2, so both lookups fetch the same row and both come back as the vector 1.2753 -0.2010 -0.1606. The values depend on the ID alone. The position of each token leaves no trace in the numbers the lookup returns.

The same reasoning holds for any token ID at any position. If the sentence were reordered so that the two fox tokens sit next to each other, the embedding layer would again return the same vectors, arranged differently. The layer produces no distinct signal for the change in order.

The self-attention mechanism relates tokens to one another inside an LLM. On its own, however, it has no notion of where a token falls in a sequence or what order the tokens appear in. Self-attention receives the vectors produced by the embedding layer. When identical IDs produce identical vectors, self-attention receives no built-in signal that separates the first occurrence of a word from the second.

The two columns above contain the same token vectors. Each word has the same embedding in both orders, because the lookup depends only on the token ID. Only the arrangement differs. Without a position signal, the mechanism receives identical material in both cases.

Consider the two sentences dog bites man and man bites dog. They use the same words, and each word has the same embedding vector in both sentences. If an LLM read only token embeddings, the two sentences would arrive at self-attention as the same set of vectors. Everything that distinguishes who bit whom would have to come from somewhere other than the token vectors.

This is why the model needs an additional source of position information. Position is the only remaining signal that can tell the first occurrence of a repeated token from the second, and that tell the model which word came first in each sentence. Once position information is added, the same vectors can be read in a meaningful order.