Self-Attention in Transformers

Imagine translating a sentence one word at a time while covering the original with your hand. Each time you need to choose the next word, you have only your memory of what you have read so far. Now imagine being allowed to lift your hand and glance back at any part of the original sentence whenever you need it. Attention mechanisms give neural networks that ability. This article introduces the idea, starting with the Bahdanau attention mechanism and ending with the link to the transformer.

As you read, keep one question in mind: when the network produces one output token, which input tokens does it actually use, and how much does each one matter? Answering that question is the thread running through every section below.

Read a sentence and you constantly decide which other words matter for understanding a given word. A pronoun such as 'it' is meaningless until you connect it to something earlier in the text. A word near the end of a long passage may depend on a phrase introduced many words before it. Language is full of these links, and they do not always sit next to each other.

The important consequence is that importance is not a fixed property of a word. Some neighbours carry most of the meaning, some carry little, and some far-away tokens carry a crucial clue. How much a word matters depends on which word is being interpreted at that moment.

An encoder-decoder RNN reads an input sequence, and the decoder then generates an output sequence. The Bahdanau mechanism keeps that overall structure but changes what the decoder is allowed to look at. The key word is selectively. At each decoding step, the decoder can access different parts of the input, and it does not have to rely on the same part every time. The mechanism was developed for RNNs in 2014 and is named after the first author of the paper that introduced it.

Without this change, the decoder is limited to what the encoder has carried forward from the input. With Bahdanau attention, the input tokens stay available for every output token. What changes from step to step is how strongly the decoder connects to each of them. The decoder therefore gets a fresh weighting of the input at every step instead of one unchanging picture of the whole sentence.

The strength of each connection is controlled by attention weights. These are the weights that determine the importance of input tokens for generating a given output token. The source defers the details of how they are computed, so the important point for now is what they mean. A larger weight means an input token matters more for the output token currently being produced. A smaller weight means it matters less.

Weights belong to a pair of positions, not to a single input token. The word 'du' may matter a great deal when generating one output token and very little when generating another. That is why the decoder must make its choice again at each step. Reading the weights for one output token gives a row of values, one per input position, and the row changes when the output token changes.

Transformers and large language models (LLMs) are built around a component called the self-attention mechanism, which is designed to handle the problem of linking words that depend on one another. Its job is to let the model weigh the importance of different words or tokens in a sequence relative to each other. Because the weighing happens between the words of the input itself, the mechanism is described as self-attention.

Self-attention is what allows a model to relate every position in an input to other positions, rather than reading each word in isolation. Where Bahdanau attention connects decoder positions to encoder positions, self-attention connects positions within one sequence. Each token is judged against the other tokens in the same input, and the judgement determines how much each neighbour should influence that token's representation.

Notice what this picture does and does not claim. It shows the kind of judgement self-attention is meant to make: deciding which parts of the input bear on which other parts. It does not show how the model computes those judgements. The source introduces the mechanism at this conceptual level and says nothing more specific about the arithmetic inside it.

A transformer is a deep learning model architecture for language translation. In the simplified depiction the source gives, it is made of two parts: an encoder and a decoder. Self-attention appears within this larger structure. Understanding the jobs of the two parts makes it easier to see where the attention-based weighing of words does its work.

The source identifies two closely linked benefits of self-attention. First, it enables the model to capture long-range dependencies, meaning relationships between words that sit far apart. Second, it captures contextual relationships within the input, so that a token's role is shaped by its surroundings. Because every position can be related to other positions, a distant word is not out of reach simply because other words lie between it and the current token.

The source connects these capabilities to output quality. By capturing these relationships, the mechanism enhances the model's ability to generate output that is coherent and contextually relevant. The reason self-attention matters is not only that it sees many words at once, but that it uses those words to judge what each one means in its setting.

The two mechanisms share an idea: a weighting that expresses how much one part of the data should influence the processing of another. They differ in where that weighting happens and in what architecture it lives inside. The source presents the transformer's self-attention mechanism as inspired by Bahdanau attention, and notes that it does not require RNN architectures.

This article establishes the conceptual foundation only. The source defers the detailed mechanics of self-attention to a later chapter. It also defers the computation of attention weights, so you should not yet expect to calculate them from the material here.