Encoder-Decoder Models for Translation
Because one hidden state must hold the encoded input, long sequences create a bottleneck. Everything the decoder knows about the input has to pass through that single state.
Why Short Sentences Are Easier
Recurrent neural networks, usually called RNNs, can do useful work on sequences of words. In translation tasks, a basic RNN can handle short sentences without much trouble. The difficulty begins when the input gets longer. A paragraph or a long document contains many words spread across a wide stretch of text, and the model has to connect words at the start with words near the end. This article asks one specific question: why does the simple RNN design fail to keep up as the input grows?
What do you think happens?
Which first word has more later updates to survive before the output is produced: the first word of a 3-word input or the first word of a 40-word input?
Reveal answer
The 40-word input
Every later word updates the hidden state, so the first word's influence has to survive 39 later updates in the 40-word input, compared with only 2 in the 3-word input. More updates means more chances for the early information to be overwritten or diluted, which is the core of the long-input problem.
Keep this question in mind while you read. The answer is not that RNNs are unable to process long input at all. The answer is about how information from earlier words has to travel to the point where the output is produced.
Tracing the Hidden State Word by Word
An RNN reads its input in order. At each step it takes one word and combines it with a hidden state, which is the network's running internal memory of what it has read so far. After the word is processed, the hidden state is updated. When the next word arrives, the network again combines that word with the updated hidden state. The process continues until the whole input has been read.
Read the diagram from left to right. Each arrow is one word being combined with the state, and each state box is what exists after that step. The state is replaced at every step rather than stored separately for each word. There is only one running memory, and it is rewritten as the input moves forward. This is what the hidden state means: a single summary of everything read so far, updated after each word.
The Encoder Hands Off One Vector
In an encoder-decoder setup, the encoder is the part of the network that reads the input. The decoder is the part that produces the output. Once the encoder has finished reading, the hidden state it holds is what gets handed to the decoder. The flow is therefore simple: input words enter the encoder one at a time, the encoder's hidden state accumulates them, and the final state is passed to the decoder, which uses it to generate the translation.
The diagram shows the key consequence. The first word does not get its own channel to the decoder. It can influence the output only by changing the hidden state, and that change then has to survive every later update. The same is true for every other word. The encoder has to pack the whole encoded input into one hidden state before passing it to the decoder.
Because one hidden state must hold the encoded input, long sequences create a bottleneck. Everything the decoder knows about the input has to pass through that single state.
Why the Bottleneck Hurts Long Inputs
The decoder has no direct way to look back at the earlier words. It sees one compressed representation, and that representation has to hold the meaning of everything read. As the input grows, one fixed-size state is asked to hold more and more. The short-sentence case works because few words have been read and few updates have happened before decoding begins. The long-input case is harder because early words pass through many updates before the decoder reads the state.
The limitation is not that RNNs cannot process long input at all. The limitation is that all of the input must funnel through the same fixed-size state. The longer the input, the more demanding that single state becomes.
Tracing a Short and a Long Input
Counting how many updates the first word must survive
Compare how many updates the first word must survive before the decoder reads the state, for a 3-word input and a 6-word input.
- 1Identify the first word's position
In both inputs, the first word is read first, so it starts the hidden state's history.
- 2Count the later words in the 3-word input
Words 2 and 3 each combine with the state and update it, so the first word is followed by 2 updates.
- 3Count the later words in the 6-word input
Words 2 through 6 each update the state, so the first word is followed by 5 updates.
- 4Compare the two
The first word's information must survive more updates in the longer input before the decoder sees the final state. This is the pressure the single hidden state faces as the input grows.
The 6-word input forces the first word through 5 later updates, compared with 2 in the 3-word input. The decoder receives only the final state, so the earlier word's contribution has to persist through all of them.
Mistakes When Reasoning About the Bottleneck
Common mistake Believing the decoder can look back at every input word
Why it failsThe decoder receives only the hidden state the encoder produced. It does not have direct access to earlier words.
Do this insteadThink of the hidden state as the only channel between the encoder and the decoder.
Common mistake Thinking each word gets its own stored memory
Why it failsThe RNN keeps one hidden state that is replaced after each word, not a separate record per word.
Do this insteadPicture one running state that is rewritten at every step.
Common mistake Concluding that RNNs cannot process long input at all
Why it failsThe source states the answer is not that RNNs are unable to process long input. The issue is how earlier information must travel to the point where output is produced.
Do this insteadFocus on the bottleneck: the difficulty is the single fixed-size state carrying everything.
Common mistake Assuming the problem only appears in very long documents rather than growing gradually with input length
Practice Tracing the Bottleneck
An input has five words, read in order: w1, w2, w3, w4, w5. Before the decoder begins, how many updates to the hidden state come after w1 was read, and why does that number matter for the bottleneck?
Hints
- Count how many words are read after w1, since each one updates the hidden state.
- Think about what the decoder receives: only the final hidden state, after every update has happened.
Key Takeaways
- An RNN reads input one word at a time and combines each word with a hidden state, its running memory of the input so far.
- After each word, the hidden state is replaced by an updated version rather than stored separately for each word.
- In an encoder-decoder setup, the decoder receives only the hidden state the encoder produced, not direct access to earlier words.
- Because one fixed-size hidden state must hold the whole encoded input, long sequences create a bottleneck that limits the design.
- Short sentences work well because few updates occur before decoding; long inputs force early information through many updates.