Traditional Attention in Sequence-to-Sequence Models
Self-attention computes attention weights by relating different positions within a single input sequence, while traditional attention relates elements of two different sequences.
A Sentence That Needs Its Neighbors
Read the sentence 'Your journey starts with one step.' The word 'journey' means something partly because of the words around it. A model that looks at each word in isolation has no direct way to notice that 'journey' and 'step' are connected in this sentence. Self-attention is a mechanism designed for exactly this kind of situation: it lets every position in a sequence examine the other positions in that same sequence and decide how much each one matters.
This article follows one path. You will see what the running example sets up, predict which operation produces the first numbers, and then connect those numbers to the idea of a context vector. Pay attention to which element is being compared against which, because that is where most confusion about self-attention starts.
What do you think happens?
Before any numbers appear, which operation do you expect to produce the first numbers in a self-attention computation?
Reveal answer
Dot products between a query element and each other input element
The first step of implementing self-attention is to compute intermediate values called attention scores. For a chosen query element, each score is a dot product between that query and one other input element.
Self-Attention Versus Traditional Attention
The word 'self' in self-attention names where the attention happens. The mechanism computes attention weights by relating different positions inside a single input sequence, learning how the parts of the input depend on one another. Those parts can be words in a sentence or pixels in an image.
Traditional attention works across two sequences. In a sequence-to-sequence model, for instance, the attention may connect an input sequence to an output sequence. The relationships being scored are between elements of the two different sequences, not between positions of one sequence and themselves.
| Aspect | Self-attention | Traditional attention |
|---|---|---|
| Sequences involved | One input sequence | Two different sequences |
| What gets related | Different positions within the same sequence | Elements of the two sequences |
| Typical setting | Words in a sentence or pixels in an image | Input sequence connected to an output sequence in a sequence-to-sequence model |
Self-attention and traditional attention differ in which elements are being related.
Setting Up the Running Example
The sequence used for the running example is the sentence 'Your journey starts with one step.' Each word has already been turned into a token embedding. The element x(1) corresponds to 'Your', and the second element x(2) corresponds to 'journey'. Each element is a d-dimensional embedding vector. The examples use three-dimensional embeddings so that the numbers fit on a page.
Tracing Query x(2) Into Scores
Take the element x(2), the embedding for 'journey', as the query. For this query, each attention score is a dot product between the query and one other input element. The inputs and the embedded query feed into the dot product, and the result is one score per input element. The sequence of steps is shown below: the query goes into dot products with the other elements, those dot products produce scores, and the scores lead onward to weights and then to a context vector.
Computing Scores for the Query journey
Using illustrative three-dimensional embeddings invented for this example, compute the attention scores for the query journey against three other input elements: Your, starts, and step.
- 1Choose the query
The query is the embedding for journey, written as $[0, 2, 1]$.
- 2Score against Your
With Your as $[1, 0, 1]$, the dot product is $0 \cdot 1 + 2 \cdot 0 + 1 \cdot 1 = 1$. The score is 1.
- 3Score against starts
With starts as $[1, 1, 0]$, the dot product is $0 \cdot 1 + 2 \cdot 1 + 1 \cdot 0 = 2$. The score is 2.
- 4Score against step
With step as $[0, 1, 3]$, the dot product is $0 \cdot 0 + 2 \cdot 1 + 1 \cdot 3 = 5$. The score is 5.
- 5Stop at scores
These three numbers are attention scores. They are intermediate values and are not yet the weights that decide how much each input contributes.
The scores for the query journey are 1 for Your, 2 for starts, and 5 for step. The next stage turns comparisons like these into attention weights.
Scores Versus Weights
Attention scores are intermediate values. They are not yet the final quantities that decide how much each input contributes. The attention weights are the values that determine the importance or contribution of each input element when computing a context vector. For the output z(2), those weights are the terms alpha21 through alpha2T, calculated with respect to x(2) and all other inputs.
Scores come from comparing a query with each input. Weights are what those comparisons turn into when the context vector is assembled. The source presents the step from scores to weights as the stage that follows the dot products.
Context Vectors as Enriched Embeddings
The goal of self-attention is to produce one context vector for each element of the input sequence. Each context vector starts from the element's own embedding and is combined with information drawn from the rest of the sequence. The result still has the shape of an embedding, but it now reflects the surrounding words. That is the sense in which it is 'enriched': a context vector contains information about one input element and all the other elements in the sequence.
Read the diagram from left to right. The sequence supplies all the inputs. One element, here the second one, takes the role of focus. The attention weights decide how much each input contributes, and the context vector for that focus element is the combination of all the inputs weighted in this way. The rounded numbers in the source diagram are example values that show the idea, not results to memorize.
Mistakes When Reading Attention Diagrams
Common mistake Treating self-attention as a link between two different sequences
What it looks likeReading the query journey as if it were scored against a separate output sequence.
Why it failsSelf-attention relates different positions within one input sequence. Traditional attention is the one that relates elements of two different sequences.
Do this insteadAsk first whether the elements being compared come from the same sequence.
Common mistake Calling the scores the final contributions
What it looks likeSaying that the score 5 for step means step contributes five times as much as other words.
Why it failsAttention scores are intermediate values. The attention weights are the values that determine how much each input contributes to a context vector.
Do this insteadKeep scores and weights separate, and move from scores to weights before talking about contribution.
Common mistake Believing a context vector is a different kind of object from an embedding
What it looks likeAssuming the context vector for journey is a new type of data unrelated to the embeddings.
Why it failsThe result still has the shape of an embedding, but it now reflects the surrounding words.
Do this insteadThink of a context vector as an enriched embedding for one position.
Common mistake Trusting visual similarity in a rounded table
What it looks likeConcluding that the embeddings for journey and starts are related because their rounded values look alike.
Why it failsThe source notes that rounded values may look similar purely by chance, so visual similarity is not evidence of a real relationship.
Debugging in Order
When debugging a self-attention computation, check the pieces in a fixed order: first the query, then the scores, then the weights. Confirm which element is the query before you examine any numbers, because the scores are defined relative to that query. Then check that the scores are the dot products against each other input element. Only after that, check how the weights were derived from the scores, since the weights decide how much each input contributes to the context vector.
- 1Identify the query element and its position in the sequence.
- 2Compute one dot product between the query and each other input element to get the scores.
- 3Check the scores as intermediate values before reading them as contributions.
- 4Derive the attention weights from the scores.
- 5Combine the inputs using the weights to form the context vector for the query.
Trace the Scores Yourself
Using the illustrative query $[0, 2, 1]$ for journey, compute the attention score against the input element $[2, 0, 1]$. Report the score and name the next stage that the source places after the dot products.
Hints
- A dot product multiplies matching positions and adds the results: $a_1 b_1 + a_2 b_2 + a_3 b_3$.
- Pair the first entries, then the second, then the third, before adding.
- The score is an intermediate value. The next stage is the one that turns scores into attention weights.
Keeping the Pieces Straight
- 1Self-attention relates different positions within one input sequence, while traditional attention relates elements of two different sequences.
- 2Attention scores are intermediate values, computed as dot products between a chosen query element and every other input element.
- 3Attention weights determine how much each input element contributes to a context vector.
- 4A context vector is an enriched embedding that contains information about one input element and all other elements in the sequence.
Key Takeaways
- Self-attention relates different positions within one input sequence, while traditional attention relates elements of two different sequences.
- Attention scores are intermediate values, computed as dot products between a chosen query element and every other input element.
- Attention weights determine how much each input element contributes to a context vector, and the source presents the step from scores to weights as the stage after the dot products.
- A context vector is an enriched embedding that combines information from one input element with information from all the other elements in the sequence.