Query, Key, and Value Projections

Many learners first meet the word 'weight' in a neural network course, where it means a number that training adjusts until the model works. That meaning is reasonable, and it is also the source of a common confusion once attention enters the picture. The word appears in two places that play very different roles. One is the set of weight matrices that a self-attention layer learns. The other is the attention weights that the mechanism computes while it runs. This article separates the two meanings before explaining how attention scores and weights are produced, because the rest of the computation depends on knowing which values are fixed by training and which are produced fresh for each query.

In the weight matrices of self-attention, 'weight' is short for 'weight parameters'. These are values of a neural network that are optimized during training. They are the fundamental, learned coefficients that define the network's connections, and once training has set them, they describe how the network connects its inputs to its outputs.

Attention weights are a separate quantity. They determine the extent to which a context vector depends on the different parts of the input. In other words, they express how strongly the network focuses on each input element when it builds the context vector for one query. Because they are computed fresh for each query, they are dynamic and context-specific rather than fixed coefficients.

Suppose the goal is only to compute a single context vector for one input token. It is still necessary to have key and value vectors for every input element, because all of the keys take part in computing the attention weights with respect to the query. Keys and values for the full set of inputs can be obtained together by matrix multiplication with the key and value weight matrices.

This projection is what maps each input token from its original embedding space into the space of queries, keys, and values. The source example projects six input tokens from a three-dimensional space onto a two-dimensional embedding space, so each token ends up with a two-dimensional key and a two-dimensional value. The same input representation therefore yields three different vectors, and each one is produced by its own learned weight matrix.

The second step is to compute attention scores. An attention score is computed as a dot product between a query vector and a key vector. This is the same kind of dot-product computation used in the simplified self-attention mechanism, with one new aspect: the query and the key are not the raw input elements. They are obtained by transforming the inputs through their respective weight matrices. The result is one unnormalized number that measures how well the query matches that key.

Indexing matters when you locate a score. The source computes the score for the second input token first. The query comes from that second token and is paired with the key at index 1. Python starts indexing at 0, so the second key is keys[1], not keys[2].

Rather than repeating that dot product for every key, the same results can be obtained for all keys at once by multiplying the query by the transposed key matrix. The second entry of the resulting row matches the single score computed earlier, which is a useful quick check when you write this code yourself.

Attention scores are not yet attention weights. The next step converts the scores by scaling them and then applying the softmax function. The scaling divides each score by the square root of the embedding dimension of the keys. Taking the square root is mathematically the same as raising to the power 0.5.

In this formula, $\omega_{i}$ is an unnormalized attention score, $d_{k}$ is the embedding dimension of the keys, and $w_{i}$ is the resulting attention weight for that key. The source code performs the same steps: it reads the key dimension from the shape of the keys, divides the scores by its square root, and applies softmax along the last dimension.

Softmax is what converts the scaled scores into the weights. Its output values are positive and add up to one across the dimension it is applied to. The attention weights for one query therefore form a distribution over the input tokens: larger weights mean the context vector leans more heavily on those tokens.

When a computed score or weight does not match what you expected, the source gives a fixed order for locating the problem: check the query source, the key index, the scaling step, and the softmax axis, in that order. Following the order matters because each check depends on the one before it. A wrong query source makes every later score wrong, while a softmax axis error only appears after the scores are already correct.