Attention Mechanisms in LLMs

Large language models process text as sequences of inputs. Each position in a sequence is represented by a vector, and a model needs a way to let each position draw on information from the other positions. The mechanism that does this is called attention. This article focuses on one central form of it, self-attention, and on a variant named scaled-dot product attention, which is the form used in LLMs.

The goal here is not to memorize a formula but to understand what the mechanism computes and how its pieces fit together. Once you can trace a small example by hand, the later material on attention becomes much easier to follow. We will move in small steps: first the arithmetic of a dot product, then the weighted sum that forms a context vector, then the way attention weights are obtained, and finally the trainable matrices that the LLM form adds.

A dot product takes two vectors of the same length and multiplies them element-wise, then sums the products. The result is a single number, not a vector. That single number is what makes the dot product useful for measuring how two vectors relate.

The key idea behind attention is the word context. Each input is represented by a vector. A context vector is a new representation built by combining the input vectors, where each input contributes in proportion to a weight. Inputs with larger weights influence the context vector more, and inputs with smaller weights influence it less. The result is a single vector that blends information from the whole set of inputs according to those weights.

Read the diagram from left to right. Each input vector feeds into the computation of attention weights. Those weights then combine the inputs into the context vector. The context vector is therefore a summary of all the inputs, not just one of them, and the weights determine how that summary is shaped.

In this formula, $x_j$ is the j-th input vector, $\alpha_j$ is its weight, and $c$ is the context vector. Multiplying a vector by a weight scales each of its elements, and adding the scaled vectors together produces the blend.

To form a context vector, you first need the weights. In the simplified attention mechanism, the weights are obtained by taking dot products between the input vectors. Each dot product produces one number, so the set of dot products gives one number per input. Those numbers are the attention weights used in the weighted sum.

The simplified mechanism computes weights directly from the inputs. Self-attention in LLMs adds a further ingredient: trainable weight matrices. These matrices compute intermediate transformations of the inputs, and the source names three of them: queries, values, and keys. Because the matrices are trainable, the way the inputs are transformed is learned rather than fixed by hand.

The diagram shows the inputs entering three separate trainable matrices, each producing one of the intermediate transformations. This is the step that distinguishes scaled-dot product attention from the simplified mechanism described earlier, which computes weights directly from the inputs.