Computing Attention Scores for a Query

Self-attention starts from a simple question: for one token in a sequence, how strongly is it connected to each of the other tokens? Answering it requires a number for every pair of tokens. This article builds that number, called an attention score, from scratch, and then explains how the scores are rescaled into attention weights.

The raw scores are only numbers that measure how strongly each input element relates to the query. Once you have them, rescaling them so that they add up to 1 turns them into shares of a whole, which is easier to interpret. Normalization is also a convention that helps maintain training stability in a large language model. The source treats it as a practical choice rather than a mathematical necessity for every possible use.

The process is concrete. You pick one input token to act as the query, then compare it against every input token, including itself, and record one number per comparison. The result is a list of scores with one entry per input token. Before running any code, decide what kind of list you expect: how many entries it should have, and which entry you would expect to be the most meaningful.

The query is a single vector taken from the input. In the source's example it is the second input token, selected by indexing position 1, because Python lists and tensors count from zero. Position matters because the query is the fixed reference point. Changing which position is the query changes which comparisons are made, and therefore changes every score in the list.

The loop visits every input token in order, computes a dot product between that token and the query, and stores the result in a preallocated tensor at the same index. Three moving parts matter. The line that sets query fixes the comparison target. The line that creates the score tensor reserves one slot per input token, sized by the number of rows in inputs. The loop body fills each slot with one dot product. Because the query is fixed and only the current token changes, every score answers one question: how does token i relate to the chosen query?

The source writes the same calculation two ways: as an explicit loop that accumulates products into a running total, and as a single torch.dot call. The two printed values should agree. That agreement shows that torch.dot is a shortcut for the element-wise multiply-and-sum you can write yourself. When you later debug a score, this pair is a useful check. If the manual loop and torch.dot disagree, the problem lies in the inputs you passed, not in the idea of the dot product.

A dot product is more than arithmetic. It measures similarity because it quantifies how closely two vectors are aligned. When two vectors point in roughly the same direction, their element-wise products tend to be large and of the same sign, so the sum is large. When they are less aligned, the products partly cancel or stay small, so the sum is smaller. A higher dot product therefore indicates greater alignment.

In self-attention, this reading turns a list of numbers into a statement about focus. The dot product determines the extent to which each element in a sequence attends to any other element. A large value in attn_scores_2 means the corresponding input token is considered closely related to the query, and a small value means it is considered less related. In the running example, the query sat scores 5 against itself, 2 against mat, and 1 against cat, so sat attends most strongly to itself.

The most direct normalization method divides every attention score by the sum of all the attention scores. Each weight is the score at its position divided by a shared total. Because every score is divided by the same total, the weights keep the same proportions to one another as the original scores, now scaled so they add to 1.

The source recommends softmax as the more common and advisable choice. Softmax first applies the exponential function to each score, then divides each result by the sum of all those exponentials. Because each output is an exponential divided by a total of exponentials, every weight is positive. The output can be read as relative importance, where a higher weight signals greater importance. Softmax is also described as better at managing extreme values.

Applying softmax to the scores 1, 5, and 2 gives weights of about 0.017, 0.936, and 0.047. The dominant score now takes most of the weight, which is a sharper contrast than simple division gave (0.125, 0.625, 0.25). Both results sum to 1, but they are not the same distribution. This is the difference the predict-and-reveal question pointed to earlier.

Because of the stability concern, the source advises using PyTorch's softmax implementation in practice. It is extensively optimized for performance and avoids the instability of the hand-written version. The call takes the score tensor and a dim argument that selects the dimension to normalize along. In the source, the call uses dim=0 on the one-dimensional score tensor.

On this small example the built-in call and the naive function give the same weights, so the practical difference is not visible here. The reason to prefer the built-in version is robustness across inputs, which a small example does not test.

When a score disagrees with expectations, trace the query assignment, the loop index, and the dot product, in that order, to find the failure. Suppose the loop is correct, but a score for position 2 is printed as 1 instead of 2. Work through the steps in order: first confirm the query was set to inputs[1]; then confirm that the loop writes to index i for the token being compared; finally recompute torch.dot for that token by hand. The first step that does not match your expectation is where the error lies.