Computing Attention Weights Step by Step
Self-attention with trainable weights keeps the basic idea of computing context vectors as weighted sums over input vectors, with only slight differences.
Starting From the Basic Mechanism
Imagine you already understand the basic self-attention mechanism. For one chosen input element, it produces a context vector by taking a weighted sum over all the input vectors. Each input vector contributes to that context vector, and the contributions are combined according to weights. The goal does not change in the version you are about to study. The question stays the same: for this particular input element, how do we blend the whole input together into one useful vector?
What does change is the set of ingredients that go into producing that blend. The trainable version adds trainable weight matrices to the computation. Keep that single addition in mind, because most of what follows is about why it matters and what it allows the model to do.
Tracing One Context Vector
Take a short sequence of three input elements, called first, second, and third. Suppose we want the context vector for the second element. The mechanism looks at all three input vectors, because a context vector is a weighted sum over the inputs. It is not limited to the chosen element alone. The second element is the focus, which is why we say the computation is specific to a certain input element.
What do you think happens?
When building the context vector for the second element, does the computation use only the second input vector?
Reveal answer
No, all three input vectors contribute
The second element is the focus of the computation, but a context vector is a weighted sum over all the inputs. The chosen element does not act alone.
Read the diagram as a flow of information rather than a sequence of code lines. The input vectors enter, the trainable weight matrices govern how they are combined, and the output is a context vector for the chosen element. The only node that carries the word trainable is the weight matrices node, and that is the piece that differs from the basic version.
The Role of Trainable Weight Matrices
The most notable difference between the two versions is the introduction of weight matrices that are updated during model training. The computation pattern stays familiar. Some of its parameters, however, are no longer fixed by the programmer. They are values the training process is allowed to adjust.
The source states the purpose of these matrices directly: they are crucial so that the attention module inside the model can learn to produce good context vectors. The word good carries the weight here. A context vector is only as useful as the blend it represents, and a blend built from fixed rules cannot improve on its own. Making the matrices trainable gives the model a way to shift that blend as it learns from data.
Basic Versus Trainable Side by Side
The source describes the difference between the two versions as slight. Both compute context vectors as weighted sums over inputs. The trainable version keeps that structure and inserts learnable parameters into it. The comparison below lays out what each version uses.
Why the Matrices Get Updated
The weight matrices are updated during model training so that the attention module can learn to produce good context vectors. Training is what turns the matrices from fixed ingredients into learned ones. The source does not go into the training procedure in this section, so the detail here is limited to what the source states: the matrices are trainable, and training adjusts them.
Mistakes When Reading the Trainable Version
Common mistake Thinking the context vector for the second element uses only the second input
What it looks likeReading the second element as the only input that matters for its own context vector
Why it failsA context vector is a weighted sum over all the input vectors. The chosen element is the focus, not the only contributor.
Do this insteadTrace all three input vectors, first, second, and third, when building the context vector for the second element.
Common mistake Assuming the trainable version replaces the weighted sum with something new
Why it failsThe source says the computation pattern stays familiar. The trainable version keeps the weighted-sum structure.
Do this insteadTreat the trainable weight matrices as an addition to the same blend, not as a new kind of computation.
Common mistake Treating the weight matrices as fixed rules set by the programmer
Why it failsThe most notable difference is that these matrices are updated during model training.
Do this insteadThink of the matrices as parameters the training process is allowed to adjust.
Practice Tracing the Blend
Using the three-element sequence first, second, and third, trace the context vector for the third element. Name each input vector that contributes to it, and name the step where the trainable weight matrices enter the computation.
Hints
- A context vector is a weighted sum over all the input vectors, not only the chosen one.
- The trainable weight matrices govern how the input vectors are combined before the blend is formed.
- Follow the flow in the diagram: inputs, then trainable weight matrices, then weighted sum, then context vector.
Key Takeaways
- The trainable version keeps the basic idea of computing context vectors as weighted sums over input vectors, with only slight differences.
- The most notable difference is the introduction of weight matrices that are updated during model training.
- A context vector for one input element is built from all the input vectors, combined according to weights, and the trainable weight matrices govern that combination.
- The trainable weight matrices are crucial so that the attention module can learn to produce good context vectors.