Text Completion with Language Models
The GPT name points to a pretraining approach introduced by Radford et al. at OpenAI. The family's later members are built by changing this original model in specific ways.
A Narrow Goal With a Wide Reach
Imagine a system whose only training goal is to guess the next word in a piece of text. That sounds like a narrow skill. Yet the GPT family of models can be pointed at a surprisingly wide range of jobs. This article follows the family from its first version to the models used in ChatGPT, and asks two questions: how can a model built around such a plain objective become useful for so much more, and how did the family grow from its first version into the models used in ChatGPT?
Where the GPT Family Begins
GPT stands for Generative Pre-Training. It was introduced by Radford et al. from OpenAI in a paper titled Improving Language Understanding by Generative Pre-Training.
That paper is the starting point of the family. Every later model in this article, including GPT-3, InstructGPT, and the model offered in ChatGPT, is connected back to this original GPT. When you see the name GPT in a later model, the important question is how that model relates to this first one.
The GPT name points to a pretraining approach introduced by Radford et al. at OpenAI. The family's later members are built by changing this original model in specific ways.
The Next-Word Prediction Task
GPT models are pretrained on a relatively simple next-word prediction task. The model is given a passage of text and must predict the word that comes next. Pretraining means this learning happens before any later specialization. The source describes the task as simple, and that simplicity is what makes its later results notable.
The diagram below shows the shape of this task as a repeating cycle over a passage. The model reads the words so far, predicts the next word, and that prediction is checked against the word that actually follows in the text. The cycle then moves on to the next position in the passage.
Tracing a short passage (illustrative)
Take the passage 'The cat sat on the mat.' and consider the position where the model has seen 'The cat sat on the' and must predict the next word.
- 1Read the context
The model is given 'The cat sat on the' as the text so far.
- 2Predict the next word
The model must output the word that comes next. A well-trained model would be expected to favor a word such as 'mat' here.
- 3Compare with the text
The word that actually follows in the passage is 'mat'. That is the word the prediction is checked against.
- 4Repeat at other positions
The same task applies at every position in the passage, so the model practices predicting many different next words.
The single objective, guess the next word, is applied again and again across text. The source does not describe the exact training mechanics beyond this task.
Tracing One Prediction Before Revealing It
What do you think happens?
A model was pretrained only to predict the next word. Can it also carry out a task such as language translation?
Reveal answer
Yes, the source says GPT models can carry out tasks such as spelling correction, classification, and language translation, even though they are pretrained on next-word prediction.
The source presents this as the remarkable part: a simple pretraining objective yields abilities beyond simple completion.
Why a Plain Objective Goes Further
The same pretrained model can be used as a text completion model. The source notes that these models are competent at completing text and can also carry out other tasks, including spelling correction, classification, and language translation. A passage that is completed correctly has to reflect a good sense of how words and sentences fit together, and the source presents the resulting range of abilities as the notable outcome of the simple pretraining objective.
From GPT to GPT-3: Scaling Up
GPT-3 is a scaled-up version of the original model. The source describes scaling along two dimensions. First, GPT-3 has more parameters than the original GPT. Second, GPT-3 was trained on a larger dataset than the original GPT.
| Aspect | Original GPT | GPT-3 |
|---|---|---|
| Relationship | The original model in the family | A scaled-up version of the original |
| Parameters | Baseline amount | More parameters than the original |
| Training dataset | Baseline size | Larger dataset than the original |
| Exact numbers | Not given in the source | Not given in the source |
The source gives the direction of change, not exact parameter counts or dataset sizes.
From GPT-3 to ChatGPT Through InstructGPT
The next link in the chain is how GPT-3 became the model offered in ChatGPT. According to the source, the original model offered in ChatGPT was created by fine-tuning GPT-3 on a large instruction dataset. The fine-tuning followed a method from OpenAI's InstructGPT paper.
Read this as a two-stage story. GPT-3 is the base that already exists after scaling. The InstructGPT method describes how that base is adjusted, using a large instruction dataset, to produce the model that appeared in ChatGPT. The source names the method and the type of dataset, but it does not describe the internal details of the fine-tuning procedure, so this article does not go beyond those facts.
Mistakes Learners Make With the GPT Story
Common mistake Treating GPT-3 as a completely different kind of model from the original GPT
Why it failsThe source describes GPT-3 as a scaled-up version of the original, with more parameters and a larger training dataset.
Do this insteadThink of GPT-3 as the same design with greater capacity and more training text.
Common mistake Assuming the source gives exact parameter counts or dataset sizes
Why it failsThe source gives only the direction of change, not specific numbers.
Do this insteadRemember the comparison as more parameters and a larger dataset, without inventing figures.
Common mistake Believing the ChatGPT model was trained from scratch on instructions
Why it failsThe source says the ChatGPT model was created by fine-tuning GPT-3 on a large instruction dataset.
Do this insteadConnect the ChatGPT model to GPT-3 through InstructGPT's fine-tuning method.
Common mistake Thinking a next-word objective can only produce text continuation
Why it failsThe source reports that these models can also carry out spelling correction, classification, and language translation.
Do this insteadTreat the wider range of tasks as the notable result of the simple pretraining objective.
Checking Your Understanding
Without looking back, explain in your own words how GPT-3 relates to the original GPT and how the model offered in ChatGPT relates to GPT-3. Name the method and the type of dataset involved in the second link.
Hints
- Start with the two dimensions the source uses to describe scaling.
- The second link runs through the InstructGPT paper's fine-tuning method.
A friend says that because GPT models are trained to predict the next word, they can only ever finish sentences. Using the source, write a short reply that corrects this claim and names at least two other tasks the source lists.
Hints
- The source lists spelling correction, classification, and language translation.
- Explain why the result is described as surprising.
Key Takeaways
- GPT stands for Generative Pre-Training and was introduced by Radford et al. from OpenAI in the paper Improving Language Understanding by Generative Pre-Training.
- GPT-3 is a scaled-up version of the original GPT, with more parameters and a larger training dataset; the source does not give exact figures.
- The model offered in ChatGPT was created by fine-tuning GPT-3 on a large instruction dataset, using a method from OpenAI's InstructGPT paper.
- GPT models are pretrained on next-word prediction, a simple task, yet they can also carry out spelling correction, classification, and language translation.