Implementing Self-Attention with Trainable Weights in Python

Imagine you already understand the basic self-attention mechanism. For one chosen input element, it produces a context vector by taking a weighted sum over all the input vectors. Each input vector contributes to that context vector, and the contributions are combined according to weights. The version you are about to study keeps that goal. The question stays the same: for this particular input element, how do we blend the whole input together into one useful vector?

What changes is the set of ingredients that go into producing that blend. The trainable version adds trainable weight matrices to the computation. Most of what follows in this article is about why that single addition matters and what it allows the model to do.

The source describes the difference between the two versions as slight. The basic mechanism already computed context vectors as weighted sums over inputs, and the trainable version keeps that structure. The computation pattern stays familiar. What changes is that some of its parameters are no longer fixed by the programmer. They become values the training process is allowed to adjust.

The most notable difference is the introduction of weight matrices that are updated during model training. The contrast in the diagram is intentional: the trainable node is the only element that differs from the basic version, and it is the piece that lets the blend be learned rather than specified in advance.

Take a short sequence of three input elements, called first, second, and third. Suppose we want the context vector for the second element. The mechanism looks at all three input vectors, because a context vector is a weighted sum over the inputs. It is not limited to the chosen element alone. The second element is the focus, which is why we say the computation is specific to a certain input element.

Read the diagram as a flow of information rather than a sequence of code lines. The input vectors enter, the trainable weight matrices govern how they are combined, and the output is a context vector for the chosen element. The only node that carries the word trainable is the weight matrices node, which is the piece that differs from the basic version.

The source states the purpose of the weight matrices directly: they are crucial so that the attention module inside the model can learn to produce good context vectors. The word good carries the weight here. A context vector is only as useful as the blend it represents. Because the matrices are trainable, the model has a way to shift that blend as it learns from data, rather than being stuck with a blend chosen in advance.

In other words, the computation pattern stays familiar, but some of its parameters are values the training process is allowed to adjust. Each context vector still comes from a weighted sum over inputs. The trainable matrices control how that sum is shaped.

The source states that the weight matrices are updated during model training. The reason follows from their purpose: the model should learn to produce good context vectors, and a fixed blend cannot be improved by the model itself. Updating the matrices is how training changes the blend so that later forward passes produce different context vectors than earlier ones.

The loop shows the logic in order. A forward pass produces context vectors. Training adjusts the trainable weight matrices. The next forward pass then runs through the adjusted matrices, so the blend it produces can differ. This source section does not spell out the exact update rule, so the description here stays at the level of what the source states: the matrices are updated during training, and that is what lets the model learn better context vectors.