Attention Weights

An attention weight tells you how strongly one input token should influence a chosen query token. On their own, though, attention weights are only numbers, and for a given query they sum to 1.0. A model cannot do much with a list of percentages. The step that turns them into something usable is the context vector: a single vector that blends the entire input sequence according to those weights.

In this article you will first run the computation that produces a context vector and trace why its output looks the way it does. Then you will see how the same procedure is repeated to produce a context vector for every token. Finally you will study dropout, a second kind of masking that is applied to the attention weight matrix during training, and learn how to check whether a dropout result is what you expect.

Write the context vector for the second token as z superscript 2. Its definition is a weighted sum over all T input vectors, where the weight is the attention that token 2 assigns to each input. In TeX form: $z^{2}=\sum_{i=1}^{T} a^{2}_{i}\, x^{i}$. Here x superscript i is the i-th input vector and a superscript 2 with subscript i is the attention weight that token 2 gives to input i. Every input contributes, including inputs that receive a very small weight.

Read this definition as a recipe with three ingredients. You need the full list of input vectors, the attention weight attached to each input for one chosen query, and a running total into which the weighted vectors are added. The order of the inputs does not change the recipe, with one requirement: each weight must stay paired with its own input vector.

The implementation follows the formula directly. It starts from a zero vector with the same shape as the query. Then it walks through the inputs one at a time. On each pass it scales the current input vector by its matching weight and adds the result into the running vector. Using enumerate, the index i selects the correct weight for the input at the same position. The zero start matters: the running vector must begin empty so that the first addition does not carry leftover values from anywhere else.

The example below uses one-hot inputs, meaning each input vector has a single 1 in its own slot. This makes each weight easy to see. With realistic embedding vectors every output value mixes contributions from all inputs, so the printed numbers look less tidy, but the mechanism is identical.

Computing one context vector answers a single question: how should one token's representation be blended from the whole sequence? A model needs an answer for every token. The procedure does not change. Attention weights are first computed for every input token acting as a query, which produces a full table with one row per query. Each row is then used in the same weighted sum to produce that query's context vector.

So the same three steps repeat for every token: take the row of weights for that query, pair each weight with its input, and add the scaled inputs into a zero-started running vector. Producing the results one after another is the direct way to do this. A matrix formulation can perform the same work in one organized pass, but the underlying three steps stay the same.

Causal attention already restricts which tokens a model may look at: each position can draw information only from itself and earlier positions. That restriction is enforced with a mask that sets disallowed weights to zero. Dropout adds a second masking step on top of it. Instead of following a fixed rule about which positions are hidden, dropout chooses positions at random and zeroes them out. Its purpose is to reduce overfitting during training.

Dropout works as a two-part operation. First it generates a mask: a pattern of positions to drop, chosen at random. Second it applies that pattern to the matrix, zeroing the selected positions. Positions that are not dropped keep their values, although they are rescaled as described in the next section. Because the mask is random, the zeros do not fall in any fixed place you could predict from the rate alone.

Dropout is applied after the causal mask, so the causal zeros remain in place while additional weights are zeroed. The causal zeros are not undone by dropout.

Zeroing half the entries would shrink the total influence of the attention weights if nothing else changed. To compensate, each surviving value is scaled up by the factor 1/dropout_rate. At a dropout rate of 0.5 the factor is $\frac{1}{0.5} = 2$, so every surviving value is multiplied by 2. This keeps the average influence of the attention mechanism consistent between training and inference.

A surviving weight of 0.4 becomes 0.8, and a surviving weight of 0.2 becomes 0.4. The values that were dropped become exactly zero, and their original magnitude does not return. When you read a printed dropout result, the surviving entries will look larger than the originals, and that enlargement is the scaling step at work rather than an error.