The Transformer Architecture

Imagine translating a sentence one word at a time while covering the original with your hand. Each time you need to choose the next word of the translation, you only have your memory of what you have read so far. Now imagine being allowed to lift your hand and glance back at any part of the original sentence whenever you need it. Attention mechanisms give neural networks exactly that ability.

This article follows that idea in two steps. It starts with the Bahdanau attention mechanism, which was developed for recurrent neural networks, and then links it to the transformer architecture. Along the way it also answers a question that often gets skipped: where do Large Language Models come from in terms of design? As you read, keep one question in mind. When the network produces one output token, which input tokens does it actually use, and how much does each one matter?

An encoder-decoder RNN has two parts. The encoder reads an input sequence, and the decoder then generates an output sequence. A classic encoder-decoder setup compresses everything the encoder read into a single fixed context vector, and the decoder relies on that one summary while it generates every output token. Trace what the decoder has available: one summary of the whole input, handed over once, and then reused at every step.

The Bahdanau mechanism keeps that overall structure, with an encoder reading the input and a decoder generating the output. What it changes is what the decoder is allowed to look at. At each decoding step, the decoder can access different parts of the input, and it does not have to rely on the same part every time. The key word is selectively. The decoder chooses which input positions to draw on for the token it is producing right now.

Read the second half of that diagram as a sequence of separate decisions. Before the change, the decoder receives one context and reuses it. After the change, a fresh context is formed for each output step, and it depends on which input positions the decoder chooses to weight at that step. The input tokens remain available the whole time. What differs is how strongly the decoder attends to each of them.

The mechanism is controlled by attention weights. The source defines them as weights that determine the importance of input tokens for generating a given output token. A larger weight means an input token matters more for the output token currently being produced, and a smaller weight means it matters less. The source defers the details of how these weights are computed, so for now the important point is their meaning: they express relative importance, and they are tied to a specific output position.

The source presents a diagram of a decoder that translates a short German phrase into English. Each input token is connected to the output token being generated, and the width of each connecting line is proportional to how important that input token is for that output token. The diagram focuses on the second output token. The same input words are available for every output token, but their connections differ in strength.

The rebuilt diagram below uses a made-up three-word sentence, not the source's German example, to show the same idea. The output token is chat, and each input token receives its own weight. The weights are illustrative numbers chosen for this lesson. They sum to one, which is one common way of making weights comparable, but the source does not state that property, so treat it as an assumption of this example only.

The main idea to hold onto is that importance belongs to a pair: one input position and one output position. The word du in the source's German example may matter a great deal when one output token is being generated and very little when another is. That is the reason attention is recomputed at each decoding step rather than set once.

Only three years after the Bahdanau mechanism, researchers found that RNN architectures are not required for building deep neural networks for natural language processing. They proposed the original transformer architecture, which includes a self-attention mechanism inspired by the Bahdanau attention mechanism. The relationship is one of inspiration, not identity.

The conceptual step is this. The Bahdanau mechanism is attention between an encoder and a decoder. The transformer's self-attention mechanism was inspired by it and is used within the transformer architecture. The name self-attention points to the part that differs: the attention is applied within a sequence rather than from a separate decoder to a separate encoder. The source does not describe the internal workings of self-attention in this section, so the comparison below establishes only how the two ideas relate.

When people talk about Large Language Models, they often focus on what the models can do: answer questions, summarize text, or write code. A natural next question is where these models come from in terms of design. The course establishes one fact early: Large Language Models are derived from a specific design called the transformer architecture.

The key idea is the direction of the relationship. The transformer architecture is the starting point, and LLMs are what get derived from it. The architecture is therefore the more general design, and each LLM is built on top of it. Understanding the architecture is one route to understanding LLMs, which is why it is a sensible place to begin.

A useful way to reason about a base design is to treat it as a template that several products are built from. The template is not any one product. Each product starts from the same template and then takes its own shape. The illustration below uses this template idea to practice the reasoning. It illustrates the relationship only and does not describe how transformers are built internally.