Pre-training Language Models

Imagine you are building an application that answers legal questions. You have access to a model that has already been trained on a very large body of data. It knows a great deal about language in general, but it was not built with legal questions in mind. You want it to behave well on your task. What should you do with it?

This is the situation that the different types of finetuning are meant to address. Each type answers a slightly different version of one question: what kind of data do you have, and what do you need the model to do with it? Before comparing the types, it helps to be clear about where they all begin, which is with transfer learning.

Transfer learning means taking a model trained for one purpose and using it for a different task. Finetuning is one way to do this, but it is not the only way. A second approach is feature-based transfer. In this approach, a model is trained to extract features from the data, usually in the form of embedding vectors. Those vectors are then passed to another model that does the actual task.

The key contrast is between reusing an output and reusing a model. In feature-based transfer, the feature extractor produces representations and the downstream model consumes them. The source describes this pattern as very common in computer vision, where models trained on the ImageNet dataset were widely used to extract image features for tasks such as object detection and image segmentation.

Notice the difference in what is trained. Feature-based transfer reuses the feature extractor as a source of representations, and the new work happens in the downstream model that consumes them. Finetuning instead trains the pre-trained model further, so the weights that were learned in pre-training are themselves adjusted.

Pre-training usually relies on self-supervision, which lets a model learn from a large amount of unlabeled data. For language models, that data is typically just sequences of text that need no annotations. Finetuning then adjusts the pre-trained model so it fits a specific task. Seen this way, finetuning is part of the same training process, extended into a later stage.

This is why the term finetuning covers so much ground. The source defines it as any training that happens after pre-training. Because it is defined by position in the sequence rather than by one fixed procedure, it can take several forms.

The path from a general model to a task-ready model can pass through more than one kind of finetuning. The order matters, and the next sections explain why. Read the diagram as one continuous line of training, not as three separate projects.

Self-supervised finetuning means finetuning a model with self-supervision, using task-related data. It is also called continued pre-training. It is useful because it can happen before finetuning with expensive task-specific data. The idea is to first move the model toward the domain you care about, using data that is plentiful and inexpensive, and only then spend the cost of annotated examples.

Supervised finetuning uses high-quality annotated data to refine the model. During supervised finetuning, the model is trained on (input, output) pairs. The input can be an instruction and the output can be a response. The purpose of this stage is to align the model with human usage and preference.

The reason this stage matters is that a model trained on very large amounts of data learns a rich understanding of the world, but that knowledge may be difficult for users to draw out for their tasks. The way the model behaves may also differ from what people prefer. Supervised finetuning addresses this gap by showing the model examples of the kind of input and response that people want.

The three finetuning types differ mainly in the data they consume. The table below sets them side by side, including the training signal each one relies on.

Language models can be autoregressive or masked. An autoregressive model predicts the next token in a sequence, using the previous tokens as its context. A masked model fills in the blank, using the tokens both before and after the gap. Finetuning can target either behavior. With supervised finetuning, you can train a model to predict the next token or to fill in the blank.

Finetuning a model to fill in the blank is called infilling finetuning. The source says it is especially useful for tasks such as text editing and code debugging, since both involve changing a part of a text while keeping the surrounding material. An important point is that you can finetune a model for infilling even if it was originally pre-trained autoregressively. The pre-training objective does not lock the model into one kind of prediction.

The data a team has determines which finetuning type fits. The flowchart below asks the questions in an order that follows from the descriptions above. Because self-supervised finetuning can come before supervised finetuning, a team with both unlabeled domain text and annotated pairs may use both, in that order.

Several errors appear often when learners first apply these ideas. Each one comes from treating finetuning as a set of separate choices rather than one training sequence with data-dependent stages.