Encoder-Decoder Models for Sequence Generation

Imagine translating a sentence one word at a time while covering the original with your hand. Each time you need to choose the next word of the translation, you have only your memory of what you have read so far. Now imagine being allowed to lift your hand and glance back at any part of the original sentence whenever you need it. Attention mechanisms give neural networks that ability.

This article follows one idea through a neural sequence model. The model is an encoder-decoder RNN: an encoder reads an input sequence, and a decoder then generates an output sequence. The Bahdanau attention mechanism, introduced for RNNs in 2014 and named after the first author of the paper that proposed it, keeps this overall structure but changes what the decoder is allowed to look at. As you read, keep one question in mind: when the network produces one output token, which input tokens does it actually use, and how much does each one matter?

Before looking at the mechanism itself, trace what happens when the decoder produces a single output token. The source uses a text-generating decoder that translates a short German phrase into English. The input tokens are the German words, the output tokens are the English words, and a row of hidden states sits between them. The source's diagram focuses on the generation of the second output token.

Read the diagram as a picture of access rather than a fixed pipeline. When the decoder generates this one output token, every input token is available to it. Each input token is connected to the output token being generated, and the connections differ in strength. The diagram below shows the flow for one step: encoder states are compared with the decoder's current state, the resulting attention weights decide how much each input contributes, and the combined result informs the output token.

The Bahdanau mechanism is attention between an encoder and a decoder. It keeps the encoder-decoder RNN structure: the encoder reads the input, the decoder generates the output. What changes is the decoder's access. At each decoding step the decoder can reach different parts of the input, and it does not have to rely on the same part every time. The key word in the definition is selectively: the decoder chooses which input parts matter for the token it is producing now.

The strength of each connection is controlled by attention weights. The source defines them as weights that determine the importance of input tokens for generating a given output token. The weights are computed by the attention mechanism, and the source defers the details of that computation. For now, the important point is what the weights mean: a larger weight means an input token matters more for the output token currently being produced, and a smaller weight means it matters less.

The weight values in the worked example are illustrative numbers chosen for this lesson, not values from the source. The source's point is about meaning: a weight is a measure of importance relative to one output token, and the same input token can receive very different weights for different outputs.

The weights are not a property of the input token alone. The source uses the German word du as an example: it may matter a great deal when the decoder generates one output token and very little when it generates another. This is why the decoder must make its choice again at each step. As the decoder advances through the output sequence, the attention distribution is recomputed, so the input parts it relies on can shift.

Consider a generated two-step trace. At step one, the decoder is generating its first English word, and the weights concentrate on input token 2. At step two, the decoder generates its second word, and the weights concentrate on input token 4 instead. The input sequence is the same at both steps; only the decoder's position in the output has changed. Attention is therefore a per-step selection, not a single fixed choice made once for the whole sentence.

Only three years after the Bahdanau mechanism, researchers found that RNN architectures are not required for building deep neural networks for natural language processing. They proposed the original transformer architecture, which includes a self-attention mechanism inspired by the Bahdanau attention mechanism. The conceptual step is this: Bahdanau attention operates between an encoder and a decoder, while the transformer's self-attention is inspired by it and is used within the transformer architecture.