The Original Transformer Architecture

Imagine you are handed a diagram of the original transformer and told that GPT was built from it. The first question a careful learner asks is simple: what did the GPT designers keep, and what did they throw away? This article answers that question. Rather than memorizing a list of differences, we will follow the path a piece of text takes through each design and let the differences appear along the way.

The original transformer is built from two kinds of blocks: an encoder block and a decoder block. In the original design, these blocks were repeated six times. Depth in this sense is simply how many times the same kind of processing stack is applied one after another to the data.

GPT keeps only the decoder half. Its stack is also much deeper. The diagram below places both designs in the same frame, so you can see which parts exist in each. Read it as a comparison: the original transformer column has both blocks, while the GPT column keeps the decoder and has no encoder.

The source ties the choice to keep the decoder and discard the encoder to what GPT is designed to do. GPT is designed for unidirectional, left-to-right processing. This makes it well suited to text generation and next-word prediction. Generation is a task where the model writes forward, so a stack organized around handling text in one direction fits the job naturally.

The source gives concrete numbers for depth. The original transformer repeated its encoder and decoder blocks six times. GPT-3, which the source uses as its main example of the GPT family, has 96 transformer layers and 175 billion parameters in total. The source makes the point that architectures like GPT-3 are significantly larger than the original transformer.

Read the GPT design as a loop, not a single pass. Text enters and is preprocessed, then the decoder processes it and the output layers produce a result. That result is not the end of the story. It is fed back in as the input for the next round, so each round depends on what the previous round produced, and the process repeats.

The loop explains how GPT produces longer text. The model does not write a whole paragraph in one step. It generates text iteratively, producing one word per round, and each new word is appended to what came before before the next round begins. This is what the source means when it says the model works one word at a time.

The course establishes one fact early: Large Language Models are derived from a specific design called the transformer architecture. The direction of this relationship matters. The transformer architecture is the starting point, and LLMs are what get derived from it. The architecture is therefore the more general design, and LLMs are built on top of it. Understanding the architecture is one of the routes to understanding LLMs.

Think of the architecture as a template that several products are built from. The template is not any one product. Each product starts from the same template and then takes its own shape. The tree shows this direction: the base design sits at the top of the derivation, and the derived systems branch below it. The diagram names only the things the source establishes. It does not list internal parts of the architecture, because the available material does not describe them.

GPT-3 was introduced in 2020. By the standards of deep learning and large language model development, that is a long time ago. Yet the source argues that understanding GPT remains as relevant as ever. More recent architectures, such as Meta's Llama models, are still based on the same underlying concepts as GPT, with only minor modifications. Studying GPT therefore helps you read newer models, because the core ideas carried forward.

A base architecture and the systems derived from it are easy to over-describe. The material supports a specific set of claims: GPT uses only the decoder, GPT-3 has 96 transformer layers and 175 billion parameters, GPT generates one word per round in a loop, and LLMs are derived from the transformer architecture. The material does not describe the internal parts of the architecture, and it does not claim that every LLM is built identically.