Transformer Architecture

For understanding the architecture, what matters is the input-output shape of this step: the words so far go in, and one next word comes out.

Programming

Where Design Starts

When people talk about Large Language Models, they usually focus on what the models can do: answer questions, summarize text, or write code. A natural next question is where these models come from in terms of design. This article begins that story with one fact the course establishes early: Large Language Models are derived from a specific design called the transformer architecture.

The goal of this article is to follow the path from that base design to one well-known family of derived models, GPT. Along the way you will see the single training objective that GPT is built around, why it is described as autoregressive, and how the decoder-only shape of GPT follows from the original transformer. Throughout, the article keeps to what the available material states, and it marks where the material stops.

One Goal Behind GPT Pretraining

Large language models can seem to do many different things, and it is tempting to assume each ability is built in separately. The GPT family takes a different starting point. Its pretraining rests on one deliberately simple goal: the model is trained to predict the next word in a sentence, using the words that come before it. Everything else the model does grows out of that single prediction task.

Consider a partial sentence such as The cat sat on the. The model sees only these words, and its job is to produce a guess for the word that comes next. A plausible continuation is mat, although a model might weigh several candidates before settling on one. For understanding the architecture, what matters is the input-output shape of this step: the words so far go in, and one next word comes out.

What the model receives as input, and what single target it must produce, for the partial sentence The cat sat on the.

The source notes that this approach helps the model understand how words and phrases typically fit together in language. That understanding is described as a foundation that can be applied to various other tasks. So the next-word objective is the pretraining step, and its benefits are meant to carry over beyond that single prediction.

Autoregressive Generation as a Loop

The source explains that decoder-style models like GPT generate text by predicting one word at a time, and for this reason they are considered a type of autoregressive model. An autoregressive model is one that incorporates its previous outputs as inputs for future predictions. The important word in that definition is previous. A model that produces a word does not throw that word away. The word becomes part of what the model reads when it works out the next one.

It helps to read the cycle as a loop. The words so far feed the prediction step. The chosen word is then added to the sequence, and the longer sequence is what the next prediction reads. This loop is the mechanism behind the autoregressive label, and it is the reason GPT's word choices depend on everything that came before.

Each predicted word is appended to the sequence, and the longer sequence becomes the input to the next prediction.

Worked example

Tracing two prediction steps

Start with the sequence The cat sat. Show how two successive predictions are made, with each output fed back as input.

1
Step 1: read the starting sequence
The model reads The cat sat and predicts a next word, for example on.
2
Step 2: append the chosen word
The word on is added, giving The cat sat on.
3
Step 3: read the longer sequence
The model now reads The cat sat on, which includes the word it just produced, and predicts again, for example the.
4
Step 4: append again
The word the is added, giving The cat sat on the. Each step uses the growing sequence as its input.

Result

The sequence grows by one word per step, and every new word is chosen based on the sequence that precedes it.

Where the GPT Architecture Comes From

The source compares GPT with the original transformer architecture covered earlier in the course, and its verdict is that the general GPT architecture is relatively simple. The simplification is specific. GPT is essentially the decoder part of the original transformer, with the encoder removed. The original transformer has two named parts, and GPT takes only the decoder side, so the encoder does not appear in its design.

Part	Original transformer	GPT
Encoder	One of the two named parts	Removed
Decoder	The other of the two named parts	Kept, and serves as the entire architecture
Overall shape	Two parts together	Decoder-only design

What GPT keeps and drops from the original transformer, as described in the source

This is why it is fair to describe GPT as a decoder-only design. The decoder part is not one piece among several in GPT. It is the whole architecture. The source does not go further and describe the internal layers of that decoder, so this article does not either.

The Transformer as a Base Design

The transformer architecture is a specific design from which Large Language Models are derived. The direction of this relationship matters. The transformer architecture is the starting point, and LLMs are what get derived from it. The architecture is therefore the more general design, and LLMs are built on top of it. Understanding the architecture is one of the routes to understanding LLMs.

The transformer architecture sits at the base, and arrows show that LLMs, including GPT, are derived from it.

A useful way to reason about a base design is to imagine it as a template that several products are built from. The template is not any one product. Each product starts from the same template and then takes its own shape. The diagram deliberately names only the two things the source establishes. It does not list internal parts of the architecture, because the available material does not describe them.

Take a concrete analogy. Suppose a workshop has one master pattern for a wooden chair. Several finished chairs are cut from that pattern, and each takes its own color, cushion, and height. The pattern is not a chair you can sit in, and no two chairs are required to be identical. This analogy only illustrates the direction of the relationship from base design to derived products. It is not a description of how transformers are built internally.

Mistakes Beginners Make

Common mistake
Treating the transformer architecture as the same thing as one trained model
Why it fails
The source says the architecture is a design, not a single trained model.
Do this instead
Keep the design (the base) separate from any particular trained system derived from it.
Common mistake
Assuming every LLM is built in exactly the same way
Why it fails
The available material supports only the claim that LLMs are derived from the transformer architecture. It does not claim that all LLMs share identical construction.
Do this instead
State the relationship at the level it was made: LLMs are derived from the transformer architecture.
Common mistake
Thinking GPT includes the encoder from the original transformer
Why it fails
The source says GPT is essentially the decoder part of the original transformer, with the encoder removed.
Do this instead
Describe GPT as a decoder-only design.
Common mistake
Believing a generated word is discarded after it is produced
Why it fails
In an autoregressive model the previous output is incorporated as input for future predictions.
Do this instead
Append each chosen word to the sequence, and read the longer sequence for the next prediction.

What the Source Establishes and What It Leaves Open

A good habit when studying a derived system is to separate what a source has actually stated from what a reader might assume. The material in this article rests on a few clearly stated claims, and it is important to know where those claims end.

Established: Large Language Models are derived from the transformer architecture, and the transformer architecture is the base design.
Established: GPT is pretrained on predicting the next word from the words before it, and it is essentially the decoder part of the original transformer without the encoder.
Established: GPT is autoregressive, because it uses its previous outputs as inputs for future predictions.
Not established here: the internal layers and components of the decoder, and how they compute each prediction.
Not established here: how other LLMs are built internally, or whether every LLM follows the same construction as GPT.

Practice Tracing the Loop

EASY

Start with the sequence The dog ran. Write out the next two steps of the autoregressive loop. For each step, state what the model reads, what word it predicts, and what the sequence looks like afterward. You may choose any plausible next words.

Hints

Each step reads the whole current sequence, including words the model produced itself.
After a prediction, append the chosen word before the next step begins.
Check that your second step reads a longer sequence than your first step did.

Your attempt (not saved)

MEDIUM

Explain in two or three sentences why GPT's design can be described as decoder-only, and why the transformer architecture is considered the base rather than one of the derived systems.