Pretraining and Fine-Tuning Language Models

GPT is a family of language models, and this article follows that family from its first version to the models used in ChatGPT. The family starts with one named paper. GPT stands for Generative Pre-Training, and it was introduced by Radford et al. from OpenAI in a paper titled Improving Language Understanding by Generative Pre-Training. That paper is the starting point of the family. Every later model in this article is described relative to it.

The word pre-training in that title points to the central idea of this article. A model is first trained on a general task, and later it is adjusted for more specific work. The first stage is called pretraining, and the adjustment stage is called fine-tuning. Pretraining happens before any later specialization, so it comes first in the story.

Imagine a system whose only training goal is to guess the next word in a piece of text. That sounds like a narrow skill. Yet the GPT models can be pointed at a surprisingly wide range of jobs, including spelling correction, classification, and language translation. The question this article answers is how a model built around such a plain objective ends up being useful for so much more.

GPT models are pretrained on a relatively simple next-word prediction task. The model is given a passage of text and must predict the word that comes next. The source describes this task as simple, and that simplicity is what makes its later results notable. The same pretrained model can then be used as a text completion model, because completing a passage is a direct extension of guessing the next word.

A single passage yields many training examples. The model does not see the whole passage as one question. Instead, it reads a window of text, predicts the word after that window, and then the window moves one word forward. The diagram below traces this movement on a short sentence. The sentence used here is an invented illustration, not a passage from the source.

GPT-3 is a scaled-up version of the original model. The source describes scaling along two dimensions. First, GPT-3 has more parameters than the original GPT. Second, GPT-3 was trained on a larger dataset than the original GPT. The source does not give specific parameter counts or dataset sizes, so the useful takeaway is the direction of change rather than exact numbers.

A helpful way to hold this in mind is to treat GPT-3 as the same design with a bigger capacity to learn and more text to learn from. The source does not describe GPT-3 as a different kind of system. That is why it introduces GPT-3 as a scaled-up version of the original model rather than as a new invention.

The next link in the chain is how GPT-3 became the model offered in ChatGPT. According to the source, the original model offered in ChatGPT was created by fine-tuning GPT-3 on a large instruction dataset. The fine-tuning followed a method from OpenAI's InstructGPT paper. Fine-tuning is the adjustment stage that comes after pretraining, so the chain runs from pretraining in the original GPT, through scaling in GPT-3, to fine-tuning that produces the ChatGPT model.

Read this as a two-stage story after scaling. GPT-3 is the base that already exists. InstructGPT describes how that base is adjusted, using a large instruction dataset, to produce the model that appeared in ChatGPT. The source names the method and the dataset type. It does not describe the internal details of the fine-tuning procedure, so this article does not go beyond those facts.

The source presents the central surprise plainly. A model pretrained on a simple next-word prediction task can also carry out tasks such as spelling correction, classification, and language translation. These tasks look different from one another, yet all of them are reached from the same pretrained model. The diagram below shows how the named tasks relate back to the single objective.