Implementing a Large Language Model Architecture
GPT-3 is fundamentally the same in model architecture as GPT-2; the main differences are scale and training data.
Guessing How Different Two Models Are
When people first hear that GPT-3 is much larger than GPT-2, a common assumption is that the newer model must be built in a completely different way. It is a reasonable guess, because a model that is over a hundred times larger seems like it should have a new design. This article tests that assumption against the source material and shows what actually changes between the two models.
What do you think happens?
Before reading on, make a guess: how different are the architectures of GPT-2 and GPT-3?
Reveal answer
GPT-3 uses the same architecture as GPT-2, scaled up.
The main differences are scale and training data. GPT-2 has 1.5 billion parameters and GPT-3 has 175 billion. GPT-3 is also trained on more data. The design itself is fundamentally the same.
Architecture Versus Scale
Two ideas are easy to blur together here. The architecture of a model is its design. Scale describes how much of that design there is, and what it costs to run. Two models can share exactly the same design and still differ enormously in scale. That is the relationship between GPT-2 and GPT-3.
Parameters are the adjustable values a model learns during training. A model with more parameters has more values that training can tune. The number of parameters is the main measure of scale used in this comparison.
The source describes GPT-3 as fundamentally the same in model architecture as GPT-2. The change is one of scale: GPT-2 has 1.5 billion parameters, and GPT-3 has 175 billion. GPT-3 is also trained on more data. Nothing in the source treats a different design as part of the change, so a learner who studies GPT-2 is studying the same architecture that GPT-3 uses.
The diagram shows the pattern. The architecture row does not change, because the design is shared. The rows for parameters, training data, and hardware change, because they are consequences of scale. The difference in cost comes from how big the model is, not from a different way of building it.
Comparing the Four Properties
The comparison covers four properties: parameter count, training data, weight availability, and hardware needs. The table below lists each property as the source discusses it. Read across each row to see where the two models differ.
| Property | GPT-2 | GPT-3 |
|---|---|---|
| Architecture | Same underlying design as GPT-3 | Same underlying design as GPT-2 |
| Parameter count | 1.5 billion | 175 billion |
| Training data | Less data | More data |
| Weight availability | Publicly available | Not described as publicly available in the source |
| Hardware needs | Can run on a single laptop | Requires a GPU cluster for training and inference |
GPT-2 and GPT-3 compared on the four properties discussed in the source
Of the four properties, the parameter count and the hardware needs show the largest gaps. The training data gap is real, but the source does not give figures for the amount of data, so it is best described as a qualitative difference. Weight availability is the property that matters most for a learner, and it is discussed further below.
Measuring the Parameter Gap
How many times larger is GPT-3 than GPT-2?
GPT-2 has 1.5 billion parameters and GPT-3 has 175 billion. Compute how many times more parameters GPT-3 has than GPT-2.
- 1Write the ratio
Divide the GPT-3 parameter count by the GPT-2 parameter count, using the same unit (billions) for both.
- 2Substitute the values
The ratio is $\frac{175}{1.5}$, since both counts are in billions.
- 3Simplify
$\frac{175}{1.5} \approx 117$, so GPT-3 has about 117 times as many parameters as GPT-2.
GPT-3 has about 117 times as many parameters as GPT-2, which is why the phrase 'over one hundred times as many' is accurate.
This calculation shows how large the scale change is in a single number. Even so, the ratio says nothing about the design. Both models still share one architecture. The ratio only counts the adjustable values that training can tune.
Why the Hardware Gap Is So Large
The gap in resource needs is much larger than the gap in parameter counts suggests. To make this concrete, the source cites estimates from Lambda Labs for how long training GPT-3 would take on a single GPU. On a single V100 datacenter GPU, the estimate is 355 years. On a single consumer RTX 8000 GPU, the estimate is 665 years.
The source's practical advice follows from these numbers. Training and inference both need a GPU cluster for GPT-3, while GPT-2 is manageable on a laptop. Because the architecture is shared, this cost difference comes from the scale of the model rather than from a different design.
Mistakes When Comparing GPT-2 and GPT-3
Common mistake Assuming GPT-3 must have a different architecture because it is much larger
What it looks likeA learner reads that GPT-3 has 175 billion parameters and concludes that its layers must be designed differently from GPT-2.
Why it failsThe source describes GPT-3 as fundamentally the same in model architecture as GPT-2. Scale and training data are the main differences.
Do this insteadTreat the architecture as shared, and attribute the cost and size differences to scale and training data.
Common mistake Treating parameter count as the only difference that matters
What it looks likeA learner says the models differ only in size, so there is nothing else to compare.
Why it failsThe comparison covers four properties: parameter count, training data, weight availability, and hardware needs. Each one affects what a learner can do.
Do this insteadCompare the models on all four properties, and check how each property affects whether you can work with the model.
Common mistake Assuming GPT-3 can be trained on a laptop because its architecture is the same as GPT-2
What it looks likeA learner reasons that since the design matches, the same laptop workflow should work for both models.
Why it failsShared architecture does not mean shared cost. The source says GPT-3 requires a GPU cluster, and the training time estimates for single GPUs run to centuries.
Do this insteadExpect GPT-2 to be manageable on a laptop, and expect GPT-3 to need a GPU cluster.
Common mistake Assuming GPT-3 weights are publicly available because GPT-2 weights are
Why it failsThe source states that GPT-2's weights are publicly available. It does not make the same claim for GPT-3.
Do this insteadCheck the availability of weights for each model separately instead of assuming they match.
Choosing a Model to Study
Suppose a learner wants to build an LLM from scratch and must choose one of the two models to study. The source gives two reasons GPT-2 is the better choice for this purpose. The first is that its weights are publicly available, so a learner can work with a real trained model. The second is that it can run on a single laptop, so a learner does not need a GPU cluster. GPT-3 fails both tests for a learner working alone: its hardware needs are far beyond a laptop, and the source does not describe its weights as publicly available.
A friend says they want to learn to implement an LLM and asks which model to study, GPT-2 or GPT-3. Using only the four properties from this article, write a short recommendation that names the model you would choose and gives two reasons based on weight availability and hardware needs.
Hints
- Start with the two reasons the source gives for GPT-2 and check each one against the comparison table.
- Remember that the architectures are the same, so the choice should be based on access and cost, not on design.
- A strong answer mentions a specific hardware requirement, such as a laptop or a GPU cluster.
Key Takeaways
- GPT-3 is fundamentally the same in model architecture as GPT-2. The main differences are scale and training data.
- GPT-2 has 1.5 billion parameters and GPT-3 has 175 billion. Parameters are the adjustable values a model learns during training.
- GPT-2's weights are publicly available and it can run on a single laptop. GPT-3 requires a GPU cluster for training and inference.
- Cost differences between the two models come from scale, not from a different design, so GPT-2 is the more practical model for learning to implement an LLM.