Hidden States in Neural Networks

The hidden state is the internal memory the network keeps while it works.

Programming

Why Translation Needs Two Cooperating Parts

Many language translation systems are built from two cooperating pieces rather than a single component. Before transformer models became dominant, a particularly popular way to build such a system was an encoder-decoder RNN. In this design, the work of translation is split between two halves: one half reads the input, and the other half writes the output.

This article follows one sentence through that design. The goal is to see where each piece of information goes next, from the words you feed in to the words that come out, and to understand why the hidden state is the thread that connects the two halves.

What do you think happens?

In a translation system with two halves, which half should be allowed to look at the original sentence?

Only the encoder reads the original sentence
Only the decoder reads the original sentence
Both halves read the original sentence directly

Reveal answer

Answer:

Only the encoder reads the original sentence

The decoder never reads the original sentence directly. It depends entirely on what the encoder has already processed, and the encoder never writes the translation. The handoff between the two is the central event in the system.

Two Submodules and Their Jobs

Definition

An encoder-decoder architecture divides translation between an encoder that reads the whole input and a decoder that writes the output.

The encoder is the submodule that reads. It receives the input text and processes it sequentially, one token at a time. Its job is to build up a representation of what it has read so far. The encoder never writes the translation.

The decoder is the submodule that writes. It begins from the encoder's result and generates the translated sentence one word at a time. The decoder never reads the original sentence directly. Everything it knows about the input has to arrive through the encoder.

The division of labor is the key idea. Neither submodule does the whole task alone. Reading and writing are separated, and the connection between them carries all the information about the input. Because of this, the handoff between the encoder and the decoder is the single most important event in the system.

Which submodule reads the input, which writes the output, and how are the two connected?

How the Encoder Builds Its Hidden State

The hidden state is the internal memory the network keeps while it works. It refers to the internal values held in the hidden layers of the network. In the encoder, its purpose is to build up a representation of what has been read so far.

The encoder's hidden state is revised each time a new token arrives. Read in order, the state after the first token reflects only that token. When the next token arrives, the state is updated to reflect both tokens. The aim is for the state left after the last token to capture the meaning of the entire input sentence.

How does the encoder's hidden state change as each input word is read, and what does it contain after the last word?

Notice what the diagram is saying. Each state is not a separate copy of a word. It is a revised version of the same memory, updated by the newest token. After the last token, that memory is the only result of reading the input that is passed onward.

Handoff to the Decoder

The encoder's final hidden state is the only thing the decoder receives from the encoder. This makes it a compact summary of everything the encoder read. The decoder begins by taking this final state as its starting point.

From that starting point, the decoder generates the translated sentence one word at a time. At each generation step, it also updates its own hidden state. The decoder's hidden state is meant to carry the context needed to predict the next word, so each new output is chosen with awareness of what has already been produced.

How does the decoder start, and what does it receive from the encoder and from its own previous output at each step?

Tracing the German-to-English Example

Worked example

Translating du, Kannst, mir into you, Can, help

Trace how the German input tokens du, Kannst, and mir become the English output tokens you, Can, and help, one step at a time.

1
Read the first input token
The encoder receives du and updates its hidden state. The state now reflects only this first token.
2
Read the second input token
The encoder receives Kannst and revises its hidden state so that it reflects du and Kannst together.
3
Read the last input token
The encoder receives mir and revises its hidden state once more. The resulting state is meant to capture the meaning of the whole input sentence.
4
Hand off the final state
The encoder's final hidden state passes to the decoder. This is the only information the decoder gets from the input.
5
Generate the first output word
The decoder starts from the handed-off state and produces you. It updates its own hidden state.
6
Generate the next output words
The decoder produces Can, then help, each time choosing the word with awareness of what it has already produced.

Result

The English sentence you, Can, help is produced one word at a time by the decoder, starting from the encoder's final hidden state.

In what order do the input words, encoder steps, decoder steps, and output words happen, and where does each piece of information go next?

The arrows between the output tokens matter. They show that the decoder produces you, Can, and help in sequence, not all at once. Each output word is a separate step, and the decoder's hidden state is updated at every one of those steps.

Mistakes Beginners Make with the Two Halves

Common mistake
Thinking the decoder reads the original sentence directly
What it looks like
Assuming that when the decoder writes Can, it looks back at the German word Kannst.
Why it fails
The decoder never reads the original sentence directly. It depends entirely on what the encoder has already processed.
Do this instead
Remember that the only information the decoder receives from the input is the encoder's final hidden state.
Common mistake
Thinking the encoder writes the translation
What it looks like
Believing the encoder outputs English words as it reads German words.
Why it fails
The encoder never writes the translation. Its job is to build a representation of what it has read.
Do this instead
Assign reading to the encoder and writing to the decoder.
Common mistake
Treating the final hidden state as just the last word
What it looks like
Assuming the handoff contains only the token mir.
Why it fails
The hidden state is revised at every step, and the state after the last token is meant to capture the entire input sentence.
Do this instead
Think of the final state as a summary built from all the tokens, not a copy of the last one.
Common mistake
Imagining all output words are produced at once
What it looks like
Picturing you, Can, and help appearing in a single step.
Why it fails
The decoder produces output one word at a time, and it updates its hidden state at each step.
Do this instead
Trace the output as a sequence of separate decoder steps.

Checking Your Understanding

MEDIUM

Without looking back, describe in your own words what the encoder's hidden state contains after the last input token, and explain why the decoder needs it before it can write the first English word.

Hints

Think about what the encoder is trying to capture as it reads each token.
Think about which piece of information the decoder receives from the encoder and nothing else.

Your attempt (not saved)

EASY

A friend says the decoder translates by looking at the German input again at every output step. Explain why this is incorrect, using the terms encoder, decoder, and hidden state.