Embedded Input Vectors

Attention weights tell us how strongly each input token should influence a chosen query token. On their own, though, they are only numbers, and they sum to 1.0. They do not yet give the model a representation it can use. The step that turns them into something usable is the context vector: a single vector that blends the whole input sequence according to those weights.

In this article you will run the computation that produces a context vector, trace the loop that does the work, and reason about why the output looks the way it does. By the end you should be able to predict the result before running anything, and to find the exact step responsible when the result is wrong.

Read the definition of a context vector as a recipe with three ingredients. You need the full list of input vectors. You need the attention weight attached to each of those inputs for one chosen query. And you need a running total into which the weighted vectors are added.

Nothing in the recipe depends on the order of the inputs, except that each weight must stay paired with its own input vector. Swapping two inputs is harmless only if their weights are swapped along with them.

In this notation, $z^{(2)}$ is the context vector for the second token, $x^{(i)}$ is the i-th input vector, and $a_{2,i}$ is the attention weight that the second token assigns to input i. The sum runs over all $T$ input tokens, so every input contributes to the result, including those that receive a very small weight.

The implementation follows the formula directly. It starts from a zero vector with the same shape as the query. Then it walks through the inputs one at a time. On each pass it scales the current input vector by its matching weight and adds the result into the running vector. Because the loop uses enumerate, the index i selects the correct weight for the input at the same position.

Consider a small generated example. Suppose there are three input vectors with two values each, x1 = [1, 2], x2 = [3, 4], and x3 = [5, 6], and the chosen query assigns them the weights 0.2, 0.5, and 0.3. The running vector starts as [0, 0], because it has the same shape as the query. Each pass scales one input by its weight and adds it in.

Notice what determines the shape of the result. The number of inputs controls how many terms the loop adds. The embedding size controls how many values each term has. The output is one vector with the embedding size, no matter how many inputs were summed. The source's own printed output for its query is three values, which you can reproduce by reasoning through the same loop.

The zero start matters. The running vector has to begin empty so that the first addition does not carry leftover values from anywhere else. Once the loop finishes, the running vector holds the complete sum, and printing it shows the context vector.

When a context vector comes out wrong, the error almost always sits in one of three places. Work through them in order: check the query index, check the normalization of the weights, and check the pairing of each weight with its input. The chain runs from scoring to normalizing weights to scaling inputs to summing, so an error at an earlier step propagates into everything after it.

Computing one context vector answers a single question: how should the second token's representation be blended from the whole sequence? A model needs an answer for every token. The source handles this by repeating the same three steps for each query rather than inventing a new method.

Attention weights are first computed for every input token acting as a query. This produces a full table with one row per query. Each row then feeds the same weighted sum, and the result is one context vector per query position.

The source states that changing the code is the only modification needed to compute all context vectors instead of only the second one. The underlying three steps remain the same: choose the weights for a query, scale each input by its matching weight, and add the results into a zero-started running total. The contexts can be produced one after another, or with a matrix formulation in a single organized pass.