Decoder-Only Language Models
Learning the decoder-only design in GPT gives you the vocabulary and the loop structure that newer language models still build on.
Tracing a Piece of Text Through GPT
Suppose you are handed a diagram of the original transformer and told that GPT was built from it. The natural first question is what the GPT designers kept and what they threw away. This article answers that question by following the path a piece of text takes through each design, rather than by memorizing a list of differences. The differences appear along the way.
What do you think happens?
Before reading on, which parts of the original transformer do you think GPT keeps?
Reveal answer
Only the decoder
GPT employs only the decoder portion of the original transformer and drops the encoder. The rest of this article explains why that choice fits the job GPT is built for.
Mapping the Original Transformer's Two Halves
The original transformer is built from two kinds of blocks: an encoder block and a decoder block. In the original design, the encoder and decoder blocks were each repeated six times. Depth in this sense is simply how many times the same kind of processing stack is applied one after another to the data.
Why GPT Keeps the Decoder and Discards the Encoder
The source ties this choice to the job GPT is built for. GPT is designed for unidirectional, left-to-right processing, which makes it well suited to text generation and next-word prediction. Generation is a task where the model writes forward, so a stack organized around processing text in one direction fits the job naturally.
Following a Text Through the Generation Loop
Read the GPT diagram as a loop, not a single pass. Text enters and is preprocessed, then the decoder processes it and the output layers produce a result. That result is not the end of the story. It is fed back in as the input for the next round. Each round therefore depends on what the previous round produced, and the process repeats.
Tracing One Round of Generation
A GPT-style model starts with the text 'The cat'. This is a hypothetical example written to illustrate the loop, not a case taken from the source.
- 1Round 1 input
The input is 'The cat'. The model produces one word, for example 'sat'.
- 2Append the word
The new word is appended to what came before, so the text is now 'The cat sat'.
- 3Round 2 input
The next round uses 'The cat sat' as its input and produces one more word, for example 'down'.
- 4Repeat
Each word is appended before the next round begins, so the text grows one word at a time.
The model writes longer text by repeating a single-word step, with every round depending on the output of the one before it.
Comparing Depth: Original Transformer and GPT-3
The original transformer uses both halves of its design, the encoder and the decoder, each repeated six times. GPT keeps only the decoder half, and its stack is much deeper. GPT-3, which the source uses as its main example of the GPT family, has 96 transformer layers and 175 billion parameters in total.
| Property | Original transformer | GPT-3 |
|---|---|---|
| Halves used | Encoder and decoder | Decoder only |
| Block repetition | Encoder and decoder blocks each repeated six times | 96 transformer layers |
| Parameters | Not stated in the source | 175 billion |
Depth comparison using the numbers given in the source
Mistakes Beginners Make About GPT's Design
Common mistake Treating layers and parameters as the same measurement
What it looks likeSaying GPT-3 is 96 parameters deep because it has 96 transformer layers and 175 billion parameters.
Why it failsLayers count the stages data passes through, while parameters are the adjustable numbers learned during training. They measure different things.
Do this insteadReport depth as layers and size as parameters, and keep the two numbers separate.
Common mistake Assuming GPT writes a whole paragraph in one step
What it looks likeBelieving a single pass through the model produces an entire paragraph.
Why it failsThe source says GPT generates text iteratively, one word per round, and each new word is appended before the next round begins.
Do this insteadDescribe generation as a loop in which each round's output feeds the next round's input.
Common mistake Concluding that GPT has no encoder idea at all
Why it failsThe source says GPT uses only the decoder portion and drops the encoder. It describes the design as the decoder half, not as a different architecture built from nothing.
Do this insteadState precisely that GPT keeps the decoder half of the original transformer and drops the encoder half.
Why GPT Still Matters for Newer Models
GPT-3 was introduced in 2020. By the standards of deep learning and large language model development, that is a long time ago. The source nonetheless argues that understanding GPT remains as relevant as ever. More recent architectures, such as Meta's Llama models, are still based on the same underlying concepts as GPT, introducing only minor modifications.
Learning the decoder-only design in GPT gives you the vocabulary and the loop structure that newer language models still build on.
Predicting the Next Round
A GPT-style model has produced the text 'The cat sat' and then produces the word 'down' in the next round. Using the loop described in this article, what does the input for the round after that contain?
Hints
- Each word produced in a round is appended to what came before before the next round begins.
- Start from the text 'The cat sat' and add 'down' at the end.
- The output of one round becomes part of the input for the next round.
Key Takeaways
- GPT employs only the decoder portion of the original transformer and drops the encoder, which fits its unidirectional, left-to-right design for text generation and next-word prediction.
- The original transformer repeated its encoder and decoder blocks six times, while GPT-3 has 96 transformer layers and 175 billion parameters.
- GPT generates text iteratively, one word per round, with each round's output appended to the input for the next round.
- GPT-3 was introduced in 2020, and newer models such as Meta's Llama are still based on the same underlying concepts, so studying GPT remains relevant.