Dot Products and Vector Similarity

Attention scores are computed by taking the dot product of one chosen query token with every input token, including itself.

Programming

Asking How Connected Each Token Is

Self-attention starts from one simple question: for one token in a sequence, how strongly is it connected to each of the other tokens? Answering that question requires a number for every pair of tokens. This article builds that number, called an attention score, from scratch. The method is concrete. You pick one input token to act as the query, compare it against every input token (including itself), and record one number per comparison. The result is a list with one score per input token.

What do you think happens?

Given the three input tokens [1, 0], [2, 2], and [0, 1], and using token 1 as the query, which token will receive the highest attention score?

Token 0
Token 1
Token 2

Reveal answer

Answer:

Token 1, with a score of 8.

Token 1 is the query itself. Its dot product with itself is 2 times 2 plus 2 times 2, which equals 8. The other two tokens each score 2, so the query's own score is the largest.

Choosing the Query Token by Position

The query is a single vector taken from the input. In the source's example it is the second input token, selected by indexing position 1, because Python lists and tensors count from zero. Position matters because the index you write decides which vector every other token gets compared against. Changing the index changes the query, and therefore changes every score in the result.

Our small example has three input tokens, each a two-number vector: token 0 is [1, 0], token 1 is [2, 2], and token 2 is [0, 1]. Writing query = inputs[1] selects the middle token, [2, 2]. The diagram below shows which token is fixed as the query and which tokens it is compared against.

Which token is the query, and how its position maps to the list of scores computed against every input token

Because the query is fixed, each score answers one question: how does this token relate to the chosen query? The query is not a separate token; it is one of the inputs, so it also gets compared with itself. That self-comparison is included on purpose.

The Dot Product as Multiply and Sum

Each call to torch.dot combines two vectors into a single number. A dot product is a concise way of multiplying two vectors element-wise and then summing the products. Element-wise means the first element of one vector is multiplied by the first element of the other, the second by the second, and so on. All those products are then added into one scalar.

Rule

The formula has three parts you can match to code. The vectors a and b have the same number of elements n. The index k walks across their positions. The sum collapses all the products into one scalar. The source writes the same calculation twice: once as an explicit loop that accumulates products into a running total, and once with the built-in torch.dot call. Both should produce the same result.

How each pair of matching components is multiplied, and how the products combine into one number

Worked example

Dot product of token 0 with the query

Compute the dot product of token 0, [1, 0], with the query, [2, 2], by hand, then check it with torch.dot.

1
Pair the elements
The first element of token 0 is 1 and the first element of the query is 2. The second element of token 0 is 0 and the second element of the query is 2.
2
Multiply each pair
1 times 2 gives 2. 0 times 2 gives 0.
3
Sum the products
2 plus 0 gives 2, which is one scalar.
4
Compare with the library
torch.dot applied to the same two vectors also returns 2. The manual loop and the library call agree.

Result

The dot product of [1, 0] and [2, 2] is 2.

The agreement between the manual calculation and torch.dot is useful beyond this example. If a manual loop and torch.dot ever disagree, the problem lies in the inputs you passed, not in the idea of the dot product.

Building the Score List with a Loop

The loop visits every input token in order, computes a dot product between that token and the fixed query, and stores the result in a preallocated tensor at the same index. The line that sets the query fixes what is being compared against everything else. The line that creates the score tensor reserves one slot per input token, sized by the number of rows in inputs. The loop body fills each slot with one dot product.

python

import torch

inputs = torch.tensor([[1.0, 0.0],
                        [2.0, 2.0],
                        [0.0, 1.0]])

query = inputs[1]
attn_scores_2 = torch.empty(inputs.shape[0])

for i, x_i in enumerate(inputs):
    attn_scores_2[i] = torch.dot(x_i, query)

print(attn_scores_2)

import torch

inputs = torch.tensor([[1.0, 0.0],
                        [2.0, 2.0],
                        [0.0, 1.0]])

query = inputs[1]
attn_scores_2 = torch.empty(inputs.shape[0])

for i, x_i in enumerate(inputs):
    attn_scores_2[i] = torch.dot(x_i, query)

print(attn_scores_2)

Output (expected)

tensor([2., 8., 2.])

Notes on the code

query = inputs[1] fixes the second token as the vector every other token is compared against.
torch.empty(inputs.shape[0]) reserves one slot per input token.
The loop changes only x_i, so each slot records how token i relates to the fixed query.

The loop pairs each input token with the same query and stores one score per token, producing the final list

Read the final list as a statement about focus. The score at index 1 is 8, which is the query's dot product with itself. The scores at indexes 0 and 2 are both 2, meaning those tokens are less closely related to the query in this example. The list has exactly one entry per input token, matching the prediction you were asked to make earlier.

Reading a Higher Score as Alignment

A dot product measures how closely two vectors are aligned. When two vectors point in roughly the same direction, their element-wise products tend to be large and of the same sign, so the sum is large. When they are less aligned, the products partly cancel or stay small, so the sum is smaller. A higher dot product therefore indicates greater alignment.

Pair of vectors	Element-wise products	Dot product	Reading
u = [1, 1] and v = [1, 1]	1, 1	2	Same direction, large positive score
u = [1, 1] and w = [1, -1]	1, -1	0	Unrelated directions, products cancel
u = [1, 1] and z = [-1, -1]	-1, -1	-2	Opposite directions, negative score

Three vector pairs compared with the dot product

In self-attention, this reading turns a list of numbers into a statement about focus. The dot product determines the extent to which each element in a sequence attends to any other element. The higher the dot product, the higher the similarity and the attention score between two elements. A large value in the score list means the input token is considered closely related to the query; a small value means it is considered less related.

Tracing Where a Score Goes Wrong

When a score disagrees with what you expect, trace three things in order: the query assignment, the loop index, and the dot product. The query is set first, so a wrong query corrupts every score. The loop then pairs each index with that query. The dot product runs last on each pair. Checking in this order tells you the first step where a wrong value appears.

python

query = inputs[0]   # wrong: meant to use position 1
attn_scores_2 = torch.empty(inputs.shape[0])

for i, x_i in enumerate(inputs):
    attn_scores_2[i] = torch.dot(x_i, query)

print(attn_scores_2)

query = inputs[0]   # wrong: meant to use position 1
attn_scores_2 = torch.empty(inputs.shape[0])

for i, x_i in enumerate(inputs):
    attn_scores_2[i] = torch.dot(x_i, query)

print(attn_scores_2)

Output (expected)

tensor([1., 2., 0.])

Notes on the code

The query line is the first place to check: it now selects token 0, [1, 0], not token 1.
The loop and the dot product are correct; they faithfully compute scores against the wrong query.
The first score is 1 instead of 2, so the divergence is traced back to the query assignment.

Comparing this output to the expected list [2, 8, 2] reveals the failure. The loop still ran three times and the dot product is still correct, so the bug is in which vector was chosen as the query. Tracing in the stated order finds it at step one, without re-examining the arithmetic.

Common mistake
Using position 0 when the intended query is the second token
What it looks like
```
query = inputs[0] when the goal is the token at position 1
```
Why it fails
Python counts from zero, so position 1 is the second token. The wrong index changes every score in the list.
Do this instead
Write query = inputs[1] for the second token and check the index before running the loop.
Common mistake
Expecting the list to skip the query token
What it looks like
Expecting only two scores for three input tokens
Why it fails
Attention scores are computed against every input token, including the query itself.
Do this instead
Expect one score per input token; the query's own score is included.
Common mistake
Treating a dot product as only a number with no meaning
What it looks like
Reading a score of 8 as just a large number
Why it fails
In attention, the score expresses how strongly a token is aligned with the query.
Do this instead
Read higher scores as greater alignment, and compare them within the same query.

Practice: Predict and Verify a Score List

MEDIUM

Using the inputs [1, 0], [2, 2], and [0, 1], set the query to the third token, [0, 1]. Before computing anything, predict which score will be largest. Then compute each dot product element by element and check your answer against the loop.

Hints

The third token is at index 2, so the query line is query = inputs[2].
For token 1, [2, 2], pair the elements with [0, 1]: 2 times 0 and 2 times 1.
Your final list should have three entries, one per input token.

Your attempt (not saved)

Key Takeaways

An attention score is the dot product of one chosen query token with every input token, including the query itself, producing one score per input token.
The query is selected by its position in the input, and zero-based indexing means inputs[1] is the second token.
A dot product multiplies two vectors element-wise and sums the products into a single scalar, which torch.dot computes in one call.
A higher dot product indicates greater alignment, so a higher attention score means the token attends more strongly to the query.
When a score is wrong, trace the query assignment, then the loop index, then the dot product, in that order, to find the first failing step.

More concepts starting with D