Next-Word Prediction and Text Generation

The core ideas of GPT, a decoder-based stack, left-to-right processing, and text produced one word at a time through a feedback loop, carry forward into newer language models. Learning GPT is therefore a way of learning the shared foundation of those models.

Programming

Starting From a Diagram

Suppose you are handed a diagram of the original transformer and told that GPT was built from it. A natural first question is what the GPT designers kept and what they threw away. That question is the goal of this article. Rather than memorizing a list of differences, we will follow the path a piece of text takes through each design and let the differences appear along the way.

Two ideas will carry the whole discussion. The first is which part of the original design GPT uses. The second is how GPT produces output, which is not a single pass over the data but a repeated loop. Both ideas connect to the task GPT is built for, which is predicting the next word and generating text.

What GPT Keeps and What It Drops

Definition

GPT employs only the decoder portion of the original transformer and drops the encoder. Everything else about the decoder-only design follows from that single choice.

The original transformer is built from two kinds of blocks: an encoder block and a decoder block. In the original design, these blocks were repeated six times. GPT keeps the decoder half of that design and does not use the encoder at all.

Why keep the decoder and discard the encoder? The source ties the choice to the job GPT is built for. GPT is designed for unidirectional, left-to-right processing, which makes it well suited to text generation and next-word prediction. Generation is a task in which the model writes forward, so a stack organized around processing text in one direction fits the job naturally.

Which parts of the original transformer does GPT keep, and which does it drop?

Measuring Depth: Six Repetitions Against 96 Layers

Depth in this sense is simply how many times the same kind of processing stack is applied one after another to the data. In the original transformer, the encoder and decoder blocks were each repeated six times. GPT-3, which the source uses as its main example of the GPT family, is far deeper and far larger. It has 96 transformer layers and 175 billion parameters in total.

Worked example

Comparing the depth of the two designs

Using only the numbers given in the source, how does the depth of GPT-3 compare with the original transformer?

1
Read the original number
The original transformer repeated its encoder and decoder blocks six times.
2
Read the GPT-3 number
GPT-3 has 96 transformer layers.
3
Compare the two
96 divided by 6 is 16, so the source's GPT-3 figure is sixteen times the original repetition count.
4
Read the comparison with care
The original count is a number of repeated blocks and the GPT-3 count is a number of layers. The source uses both to make the point that GPT-3 is much deeper, so treat the ratio as an illustration of scale rather than a precise like-for-like measurement.

Result

GPT-3 is far deeper than the original transformer: 96 layers compared with six repetitions, and it also has 175 billion parameters, which the source describes as significantly larger than the original design.

Tracing One Round of Text Generation

Read the diagram as a loop, not a single pass. Text enters and is preprocessed, then the decoder processes it and the output layers produce a result. That result is not the end of the story. It is fed back in as the input for the next round. Each round therefore depends on what the previous round produced, and the process repeats.

After the model predicts one word, what gets appended to the input, and how does the loop repeat?

The loop explains how GPT produces longer text. The model does not write a whole paragraph in one step. It generates text iteratively, producing one word per round, and each new word is appended to what came before before the next round begins. This is the meaning of the source's statement that the model works one word at a time.

What do you think happens?

After GPT produces one word in a round, what happens to that word before the next round begins?

It is discarded and the next round starts from an empty input
It is appended to what came before and fed back in as input for the next round
It is stored as the final output and the loop stops

Reveal answer

Answer:

It is appended to what came before and fed back in as input for the next round.

Each round's output feeds the next round, so the text grows one word at a time and each round depends on the previous one.

Why the Loop Is Left to Right

The left-to-right direction is part of the design, not an accident of the diagram. GPT is designed for unidirectional, left-to-right processing. Because each new word is added to the end of the existing text, the model always works from what has already been written toward what comes next.

In the real world

Think of a writer who completes a sentence and then reads back over the whole sentence before choosing the next word. Each choice builds on everything already on the page. GPT's loop follows the same forward pattern: the text so far is the input, one word is produced, and that word becomes part of the text for the following round. This is an illustrative analogy generated for this article and is not a claim about how any writer actually works.

Common Mistakes When Reading the Loop

Common mistake
Thinking GPT writes a whole paragraph in one pass
Why it fails
The source says the model generates text iteratively, producing one word per round.
Do this instead
Picture repeated rounds, each adding one word that the next round depends on.
Common mistake
Assuming GPT uses both the encoder and the decoder
Why it fails
GPT employs only the decoder portion of the original transformer and drops the encoder.
Do this instead
When you compare the designs, the encoder is the part GPT does not use.
Common mistake
Treating layers and parameters as the same number
Why it fails
Layers describe how many stages data passes through, while parameters are the adjustable numbers learned during training.
Do this instead
Report GPT-3 as 96 layers and 175 billion parameters, and keep the two measures separate.
Common mistake
Treating the diagram as a single pass from input to output
Why it fails
The result is fed back in as the input for the next round, so the process repeats.
Do this instead
Read the diagram as a loop whose output becomes part of its own next input.

Why GPT Still Matters Today

GPT-3 was introduced in 2020. By the standards of deep learning and large language model development, that is considered a long time ago. Yet the source argues that understanding GPT remains as relevant as ever. More recent architectures, such as Meta's Llama models, are still based on the same underlying concepts as GPT, introducing only minor modifications.

Key point

Practice: Trace the Generation Loop

EASY

Without looking back, describe in your own words the path one round of GPT takes from the input text to the next round. Name the stages in order and say what happens to the output of each round. Then compare your description with the loop diagram in the earlier section.

Hints

Start with the input text and follow it through preprocessing and the decoder.
Notice what the output layers produce and where that result goes next.
Check whether your answer shows that each round depends on the previous one.

Your attempt (not saved)

MEDIUM

Explain, in two or three sentences, why GPT drops the encoder. Use the phrase unidirectional, left-to-right processing in your answer and connect it to text generation.