Simplified Self-Attention Without Trainable Weights
Self-attention computes attention weights by relating different positions within a single input sequence, while traditional attention relates elements of two different sequences.
Why a Word Needs Its Neighbours
Read the sentence 'Your journey starts with one step.' The word 'journey' means something partly because of the words around it. A model that looks at each word in isolation has no direct way to notice that 'journey' and 'step' are connected in this sentence. Self-attention is a mechanism designed for exactly this situation. It lets every position in a sequence examine the other positions in that same sequence and decide how much each one matters.
What do you think happens?
Suppose you want to find out how strongly the word 'journey' relates to the other words in the sentence. How many numbers would you expect to compute for that one word, and what would each number describe?
Reveal answer
One number for each input element. Each number is a score that compares the query element 'journey' with one input element in the sequence.
The source describes attention scores as dot products between a chosen query element and every other input element, so the output is one score per input element rather than a single overall score.
This article follows one path. First you will see what the running example sets up. Then you will trace the operations that produce the first numbers, and finally connect those numbers to the idea of a context vector. Pay attention to which element is being compared against which, because that is where most confusion about self-attention starts.
The Meaning of the Word Self
Self-attention computes attention weights by relating different positions within a single input sequence. Traditional attention, by contrast, relates elements of two different sequences.
The word 'self' in self-attention names where the attention happens. The mechanism looks inside one sequence and learns how the parts of that input depend on one another. Those parts can be words in a sentence or pixels in an image. Nothing outside the sequence is being compared.
Traditional attention works across two sequences. In a sequence-to-sequence model, for instance, attention may connect an input sequence to an output sequence. The relationships being scored are between elements of the two different sequences, not between positions of one sequence and themselves.
| Aspect | Self-attention | Traditional attention |
|---|---|---|
| Sequences involved | One input sequence | Two different sequences |
| What is related | Positions within the same sequence | Elements of one sequence to elements of the other |
| Example from the source | Each word in 'Your journey starts with one step' examining the other words in that sentence | Attention connecting an input sequence to an output sequence in a sequence-to-sequence model |
| Typical output | One context vector per element of the input sequence | Relationships scored between the two sequences |
Self-attention and traditional attention differ in which elements are being compared.
Tracing the Running Example
The sequence used throughout this article is the sentence 'Your journey starts with one step.' Each word has already been turned into a token embedding. The element x^{(1)} corresponds to 'Your', and the second element x^{(2)} corresponds to 'journey'. Each element is a d-dimensional embedding vector. The examples here use three-dimensional embeddings so the numbers fit on a page.
Notice what the diagram is and is not saying. The query is journey, and the scores are formed by comparing journey with each input element in turn. The diagram is a map of which comparisons happen, not a claim about the exact numbers. The source's tables are rounded, so the printed values are approximations of what the dot products would produce.
From Dot Products to Attention Scores
The first implementation step is to compute intermediate values called attention scores, written as omega in the source. For a chosen query element, each attention score is a dot product between that query and one other input element. Each score is therefore a single number produced by comparing two embedding vectors. The query x^{(2)} is compared with every input element, which yields one score per input.
Attention scores are intermediate values. They are not yet the final quantities that decide how much each input contributes. A helpful way to keep the two apart is this: scores come from comparing a query with each input, and weights are what those comparisons become when the context vector is assembled.
Turning Scores into Attention Weights
The source presents the step from scores to weights as the stage that follows the dot products. The pipeline below shows the order of operations for one query. The raw scores are the output of the dot products, and the attention weights are the values that determine the importance or contribution of each input element when computing a context vector.
For the element z^{(2)}, the attention weights are the terms $\alpha_{21}$ through $\alpha_{2T}$. They are calculated with respect to x^{(2)} and all other inputs. Each weight answers the question of how much a particular input should count toward the output for the query.
The Context Vector as a Blend of Inputs
The goal of self-attention is to produce one context vector for each element of the input sequence. The context vector for a focus element is the combination of all the inputs, each weighted by its attention weight. Read the data flow below from left to right. The sequence supplies all the inputs, one element takes the role of focus, the weights decide how much each input contributes, and the combination is the context vector.
The result still has the shape of an embedding, but it now reflects the surrounding words. That is the sense in which the context vector is enriched. The diagram uses rounded example values only to show the idea, and those numbers are not results to memorize.
Original Embedding Versus Context Vector
Consider the element journey before and after self-attention is applied. Its original embedding describes only the word itself. Its context vector starts from that same embedding but is combined with information drawn from the rest of the sequence, including starts, one, and step. The before and after view below shows what has changed.
Mistakes Beginners Make with Self-Attention
Common mistake Treating self-attention as a comparison between two different sequences
Why it failsSelf-attention relates positions within one input sequence. Traditional attention relates elements of two separate sequences.
Do this insteadAsk first which sequence the positions belong to. If both query and inputs come from the same sequence, you are in the self-attention case.
Common mistake Using attention scores as the final contribution of each input
What it looks likeReading the dot product between journey and starts as the amount starts contributes to the context vector.
Why it failsScores are intermediate values. The attention weights are what determine the contribution of each input element.
Do this insteadFollow the order query, then scores, then weights. Only the weights decide how much each input contributes.
Common mistake Reading similar-looking rounded numbers as a real relationship
Why it failsThe source notes the figure's numbers are truncated to one digit after the decimal point, so the embeddings for journey and starts may look similar purely by chance.
Do this insteadTreat rounded tables as approximations. Judge relationships by the computation, not by visual resemblance.
Common mistake Expecting the context vector to have a different shape from the input embedding. The source says the result still has the shape of an embedding, only enriched with information from the rest of the sequence.
When debugging self-attention, check the query first, then the scores, then the weights, in that order. This follows the order in which the values are produced and makes it clear which stage has gone wrong.
Tracing a New Query
Using the running sentence 'Your journey starts with one step', choose a different query word, such as starts. Write down which element is the query, how many attention scores you would compute, which pairs of positions produce them, and which values would then be turned into attention weights before the context vector is formed.
Hints
- Start with the query element, then list the input elements it is compared against.
- Count the input elements in the sentence. Each one gets its own score from a dot product with the query.
- Remember that scores are intermediate. The next step after scores is attention weights, and only those weights set each input's contribution.
Key Takeaways
- Self-attention relates different positions within one input sequence, while traditional attention relates elements of two different sequences.
- Attention scores are intermediate dot products between a chosen query element and every other input element, one score per input element.
- Attention weights determine how much each input element contributes, and they are what the scores become when the context vector is assembled.
- A context vector is an enriched embedding: it keeps the embedding's shape but combines information from every element of the input.
- When debugging, check the query, then the scores, then the weights, in that order.