GPT Model Sizes
A transformer block is a core structural component of GPT models that combines masked multi-head attention modules with fully connected feed forward networks using the GELU activation function.
Why Start With One Repeated Unit
A GPT model is a large language model, and its internal design is built from a repeated unit called the transformer block. When people say a GPT model has many layers, they are pointing at this repetition. Before looking inside a model of this kind, it helps to ask a concrete question: if a model is made of one repeated unit, what does that unit actually do to the data passing through it?
This article answers that question in stages. First you will see what one transformer block contains. Then you will see how many of these blocks are stacked to form a GPT model, how text is produced from the model's output tensors, and why training is needed before the output becomes coherent.
Inside One Transformer Block
A transformer block is a core structural component of GPT models that combines masked multi-head attention modules with fully connected feed forward networks using the GELU activation function.
The definition names two parts. The first is the masked multi-head attention module. The second is the fully connected feed forward network, and the source specifies that this network uses the GELU activation function. Data passes through these parts in order within the block. The diagram below traces that path from an input tensor to an output tensor.
Stacking Blocks Into a GPT Model
A GPT model is an LLM with many repeated transformer blocks. Each block has the same two-part structure described above, and the model is formed by repeating that structure many times. The source describes these blocks as having millions to billions of parameters, which means the parameters of a GPT model are spread across this repeated structure rather than living in a single large component.
When you read that many layers means many blocks, keep the hierarchy in mind. The repetition is of the whole unit, so a deeper model is not a new kind of component. It is the same unit applied again and again. The diagram above shows this nesting: the model contains blocks, and each block contains the two parts.
How Model Size Is Set
GPT models come in various sizes, measured by their number of parameters. The source gives four example sizes: 124 million, 345 million, 762 million, and 1,542 million parameters. The source also notes that all of these can be implemented with the same GPTModel Python class. The size of a model is therefore set by configuration, not by writing a different class for each size.
Reading a size label
A model is described as having 124 million parameters. What does that number tell you, and what does it not tell you?
- 1Identify the measure
The source says GPT models are sized by their number of parameters, so the number 124 million counts parameters.
- 2Check the class
The source says this size, like the others, can be implemented with the same GPTModel Python class. The number does not name a different class.
- 3Connect to the structure
Parameters are spread across the repeated transformer blocks, so the size reflects how the same block structure is configured.
The label gives the parameter count of the configured model. It does not require a separate class for that size.
Tracing Text Generation Step by Step
The output of the model is a set of numeric tensors, and generation is the step that turns those numbers into text a person can read. The source describes this as decoding output tensors into human-readable text by sequentially predicting one token at a time, based on a given input context. The transformer blocks sit in the middle of this pipeline, so their outputs are what the decoding step works from.
What do you think happens?
Does a GPT model produce a whole sentence in a single step?
Reveal answer
No, it predicts one token at a time
The source says text generation is a sequential process. The model predicts a token based on the input context, and that prediction is decoded. It does not produce a complete sentence in one step.
The key phrase in the source is one token at a time. The model does not write a whole sentence in one pass. It predicts one token from the current input context, and the loop then uses that result as part of the context for the following prediction. The diagram traces this cycle, which repeats until the text is produced.
Why an Untrained Model Writes Nonsense
Having the right structure is not enough for useful output. The source states that without training, a GPT model generates incoherent text, which underscores the importance of model training for coherent text generation. The architecture defines how data flows through the blocks, but training is what makes the generated text coherent.
Common Misreadings of GPT Model Sizes
Common mistake Assuming each model size needs its own Python class
Why it failsThe source states that all of the example sizes, 124, 345, 762, and 1,542 million parameters, can be implemented with the same GPTModel Python class.
Do this insteadTreat size as a configuration choice applied to the same class.
Common mistake Thinking more layers means a different kind of block
Why it failsEach transformer block has the same two-part structure, and the model is formed by repeating that structure many times.
Do this insteadRead depth as repetition of one unit, not as a new component.
Common mistake Believing the model produces a full sentence in one step
Why it failsThe source describes text generation as sequentially predicting one token at a time from an input context.
Do this insteadThink of generation as a loop that adds one predicted token per pass.
Common mistake Expecting coherent text from an untrained model
Why it failsThe source says that without training, a GPT model generates incoherent text.
Do this insteadRecognize that training is what makes the output coherent, while the architecture defines the data flow.
Checking Your Understanding
Without looking back, write the order in which data moves through one transformer block, then explain in one or two sentences why an untrained GPT model's output is incoherent even though its structure is complete.
Hints
- The source names two parts of the block, and says the attention module comes before the feed forward network inside the block.
- Look for the sentence that separates the architecture, which defines how data flows, from training, which makes the text coherent.
Article Summary
- 1A transformer block combines a masked multi-head attention module with a fully connected feed forward network that uses the GELU activation function, and data passes through these parts in order.
- 2A GPT model is an LLM built by repeating many transformer blocks with the same structure, and its parameters are spread across that repetition.
- 3GPT model sizes, such as 124, 345, 762, and 1,542 million parameters, are set by configuration within the same GPTModel Python class.
- 4Text generation decodes output tensors into text by predicting one token at a time from an input context, and the predicted token is fed back into the context.
- 5Without training, a GPT model generates incoherent text, so training is what makes the output coherent.
At a glance
- A transformer block combines a masked multi-head attention module with a fully connected feed forward network that uses the GELU activation function.
- A GPT model is an LLM built by repeating many transformer blocks, and its parameters are spread across this repeated structure.
- GPT model sizes such as 124, 345, 762, and 1,542 million parameters are set by configuration within the same GPTModel Python class.
- Text generation decodes output tensors into human-readable text by predicting one token at a time from an input context.
- An untrained GPT model generates incoherent text, so training is what makes the output coherent.