Attention Weights and Scaled Dot-Product Attention
A causal attention mask is used when an LLM reads and generates text from left to right, to prevent access to future tokens.
Why Attention Needs Limits
Large language models process text as sequences of inputs, and they need a way to let each position in a sequence draw on information from the other positions. The mechanism that does this is called attention. Attention lets a language model decide how much each token should draw on other tokens in the sequence. That flexibility is powerful, but it creates a problem when the model is meant to generate text from left to right. If every token could freely look at every other token, the model could peek at words that have not been produced yet.
This article covers two masking techniques that place controlled limits on attention. The first is the causal attention mask, which blocks access to future tokens. The second is the dropout mask, which is used to reduce overfitting. Along the way you will also see the basic arithmetic behind attention: the dot product, the weighted sum that produces a context vector, and the trainable matrices that produce queries, keys, and values in scaled-dot product attention, the form used in LLMs.
Context Vectors as Weighted Sums
The key idea is the word context. Each input is represented by a vector. A context vector is a new representation built by combining the input vectors, where each input contributes in proportion to a weight. Inputs that receive larger weights influence the context vector more, and inputs that receive smaller weights influence it less. The result is a single vector that blends information from the whole set of inputs according to those weights.
- 1x_j is the input vector at position j
- 2w_j is the attention weight that position j receives
- 3Each input vector is scaled by its weight, and the scaled vectors are added together
- 4The sum z is the context vector: one vector that summarizes all the inputs
Building a context vector from three inputs
Three inputs are x1 = [1, 0], x2 = [0, 1], and x3 = [1, 1]. Their attention weights are 0.5, 0.2, and 0.3. Find the context vector.
- 1Scale each input by its weight
0.5 times [1, 0] gives [0.5, 0]. 0.2 times [0, 1] gives [0, 0.2]. 0.3 times [1, 1] gives [0.3, 0.3].
- 2Add the scaled vectors element by element
The first component is 0.5 + 0 + 0.3 = 0.8. The second component is 0 + 0.2 + 0.3 = 0.5.
The context vector is [0.8, 0.5]. It leans toward x1 and x3 because they received the larger weights, and it contains a little of x2.
Dot Products as the Scoring Step
A dot product multiplies two vectors element-wise and sums the products, producing a single number. Written as a formula, for vectors a and b with n components, the dot product is a · b = sum over i of a_i times b_i.
Notice that the output is a single number, not a vector. This is what makes the dot product useful for measuring how two vectors relate. The source describes it as a concise notation for the multiply-then-sum operation, so the same calculation can be written compactly wherever it appears.
What do you think happens?
What single number is the dot product of a = [1, 2, 3] and b = [4, -1, 2]?
Reveal answer
8
The element-wise products are 1 times 4 = 4, 2 times -1 = -2, and 3 times 2 = 6. Summing them gives 4 - 2 + 6 = 8.
a = [1, 2, 3]
b = [4, -1, 2]
score = sum(x * y for x, y in zip(a, b))
print(score)- 1Line 1 defines the first vector as a list of three numbers.
- 2Line 2 defines the second vector in the same way.
- 3Line 3 multiplies matching components and adds the three products, producing one number.
- 4Line 4 prints that single score.
- zip pairs up the components at the same position in a and b
- x * y multiplies each pair element-wise
- sum adds the products into one scalar
8From Simplified Attention to Trainable Queries, Keys, and Values
In a simplified attention mechanism, attention weights are computed via dot products. This connects the dot product to the weighted sum from earlier. To form a context vector, you first need the weights, and the simplified mechanism obtains them by taking dot products between the input vectors. Each dot product produces one number, so the set of dot products gives one number per input, and those numbers are the attention weights used in the weighted sum.
The simplified mechanism computes weights directly from the inputs. Self-attention mechanisms used in LLMs add a further ingredient: trainable weight matrices. These matrices compute intermediate transformations of the inputs, and the source names three of them: queries, values, and keys. Because the matrices are trainable, the way the inputs are transformed is learned rather than fixed by hand. The source pack does not go into further detail on how these roles are combined at every step, so the key point here is that the trainable matrices produce these intermediate representations from the inputs.
Read the diagram from top to bottom. The same input enters three separate trainable matrices. Each one produces its own representation: a query, a key, or a value. The queries and keys are the pieces that feed the scoring step, while the values are the representations that get blended into the output. This is the step that distinguishes scaled-dot product attention from the simplified mechanism, which computes weights directly from the inputs.
Left-to-Right Generation and the Causal Mask
Some LLMs read and generate text from left to right. The model produces one token, then uses what it has produced so far to produce the next one. At any position, the tokens that come later in the sequence are not yet available to the model in the way a finished sentence would be. The causal attention mask exists to enforce this ordering inside the attention computation, so that a token's attention can only be spread over the tokens at or before its own position.
Attention weights describe how strongly one token attends to each other token. The causal attention mask sets the weights that point toward future positions to zero, so those positions contribute nothing to the result. Positions that are not in the future remain available. The mask therefore does not change which tokens are present in the text; it changes which of them each token is permitted to use.
- Each row is the position doing the attending
- Each column is the position being attended to
- Zeros fall only in columns to the right of the diagonal, which are the future positions
Masking the weights for the token sat
Before masking, the token sat has raw attention weights of 0.2 toward The, 0.5 toward cat, 0.3 toward sat, and 0.4 toward down. Apply the causal mask.
- 1Identify the future positions for sat
sat is at position 3 of 4. The only future position is down, which is position 4.
- 2Zero the weight pointing to the future
The weight toward down changes from 0.4 to 0. The weights toward The, cat, and sat are not changed.
After masking, sat attends to The, cat, and itself only. Its visible weights are 0.2, 0.5, and 0.3, and the future token down contributes nothing.
Dropout Masks and Overfitting
A dropout mask is used in addition to causal attention masks, and its purpose is to reduce overfitting by randomly dropping out attention weights. The dropout mask answers a different question from the causal mask. It addresses how to keep the model from relying too heavily on any particular attention pattern, which is a concern about overfitting rather than about the direction of access.
The two masks are applied together rather than as alternatives. The causal attention mask first determines which positions are blocked because they lie in the future. The dropout mask then operates on the attention weights as an additional step, randomly removing some of them to help reduce overfitting. The dropout mask is layered on top of the causal masking; it is not used in its place.
Common Mistakes With Attention Masks
Common mistake Thinking the causal mask removes future tokens from the text
Why it failsThe mask does not change which tokens are present. It only sets the attention weights pointing toward future positions to zero.
Do this insteadDescribe the causal mask as limiting which tokens each position may use, not as deleting tokens.
Common mistake Treating the dropout mask as a replacement for the causal mask
What it looks likeRemoving the causal mask because dropout is already applied would let a token peek at positions that have not been generated yet.
Why it failsThe two masks answer different questions. The dropout mask is layered on top of causal masking and does not control direction of access.
Do this insteadKeep both masks. Apply the causal mask to block future positions, then apply dropout to reduce overfitting.
Common mistake Assuming a dot product returns a vector
What it looks likeComputing [1, 2, 3] dot [4, -1, 2] and expecting a list of three products.
Why it failsThe products are summed, so the result is a single number.
Do this insteadMultiply element-wise to form the products, then add them to get one scalar, which is 8 in this case.
Common mistake Confusing the simplified attention mechanism with scaled-dot product attention, which adds trainable query, key, and value matrices
Checking Your Understanding
A token at position 2 of 5 is attending over the sequence. Which positions does the causal mask zero out for that token, and which positions remain available? Explain your answer in terms of left-to-right generation.
Hints
- Positions later than the current position are the future.
- The causal mask zeroes weights pointing toward future positions only.
Compute the dot product of a = [2, 0, -1] and b = [3, 5, 4]. Then state whether the result is a vector or a single number.
Hints
- Multiply the components in matching positions first.
- Add the three products together at the end.
Summary of the Attention Pieces
- 1A dot product multiplies two vectors element-wise and sums the products into one number.
- 2A context vector is a weighted sum of the input vectors, so inputs with larger weights influence it more.
- 3The causal attention mask zeroes out attention weights pointing toward future positions, so left-to-right generation cannot peek ahead.
- 4The dropout mask is applied on top of the causal mask to reduce overfitting by randomly dropping attention weights.
- 5Scaled-dot product attention uses trainable matrices that produce queries, keys, and values from the inputs.
At a glance
- A dot product multiplies two vectors element-wise and sums the products into one number, which is how attention scores are computed in the simplified mechanism.
- A context vector is a weighted sum of the inputs, so the inputs with larger attention weights shape it more.
- The causal attention mask zeroes out weights toward future positions, so a token attends only to itself and earlier tokens during left-to-right generation.
- The dropout mask is layered on top of the causal mask to reduce overfitting by randomly dropping attention weights.
- Scaled-dot product attention adds trainable matrices that produce queries, keys, and values from each input.