Supervised Finetuning in Practice

Imagine you are building an application that answers legal questions. You have access to a model that was already trained on a very large body of data, and you want it to behave well on your task. The model knows a great deal about language in general, but it was not built with legal questions in mind. What should you do with it?

This is the situation that the different types of finetuning are meant to address. Each type answers a slightly different version of the same question: what kind of data do you have, and what do you need the model to do with it? Before comparing the types, it helps to be clear about where they all begin, which is with transfer learning.

Pre-training usually relies on self-supervision, which lets a model learn from a large amount of unlabeled data. For language models, that data is typically just sequences of text that need no annotations. Finetuning then adjusts the pre-trained model so it fits a specific task. Seen this way, finetuning is part of the same training process, extended into a later stage rather than a separate procedure.

Because finetuning can happen in several stages, the path from a general model to a task-ready model can pass through more than one kind of finetuning. A common path places a self-supervised stage first and a supervised stage after it. The order matters, and the rest of this article explains why.

Read the timeline from left to right. The training objective and the data both change at the boundaries. Pre-training works from large amounts of unlabeled text. The self-supervised finetuning stage keeps learning from unlabeled sequences, but those sequences now come from the domain you care about. The supervised stage switches to annotated (input, output) pairs. Each stage starts from the weights left by the previous one, which is why finetuning is best understood as a continuation rather than a fresh start.

Transfer learning means taking a model trained for one purpose and using it for a different task. Finetuning is one way to do this, but it is not the only way. A second approach is feature-based transfer. In this approach, a model is trained to extract features from the data, usually in the form of embedding vectors. Those vectors are then passed to another model that does the actual task.

The key contrast is between reusing an output and reusing a model. In feature-based transfer, the feature extractor produces representations and the downstream model consumes them. The source describes this pattern as very common in computer vision, where models trained on the ImageNet dataset were widely used to extract image features for tasks such as object detection and image segmentation.

Self-supervised finetuning means finetuning a model with self-supervision, using task-related data. It is also called continued pre-training. It is useful because it can happen before finetuning with expensive task-specific data. The idea is to first move the model toward the domain you care about, using data that is plentiful and inexpensive, and only then spend the cost of annotated examples.

Supervised finetuning uses high-quality annotated data to refine the model. During supervised finetuning, the model is trained on (input, output) pairs. The input can be an instruction and the output can be a response. The purpose of this stage is to align the model with human usage and preference.

The reason this stage matters is that a model trained on very large amounts of data learns a rich understanding of the world, but that knowledge may be difficult for users to draw out for their tasks. The way the model behaves may also differ from what people prefer. Supervised finetuning addresses this gap by showing the model examples of the kind of input and response that people want.

Language models can be autoregressive or masked. An autoregressive model predicts the next token in a sequence, using the previous tokens as its context. A masked model fills in the blank, using the tokens both before and after the gap. Finetuning can target either behavior. With supervised finetuning, you can train a model to predict the next token or to fill in the blank.

Finetuning a model to fill in the blank is called infilling finetuning. The source says it is especially useful for tasks such as text editing and code debugging, since both involve changing a part of a text while keeping the surrounding material. In each case the model sees the text on both sides of the part being changed, and must produce the missing span.

The three types differ mainly in the kind of data they use. Supervised finetuning uses explicit (input, output) pairs. Self-supervised finetuning uses task-related sequences of text that need no annotations. Infilling finetuning uses text in which a span is masked, so the model must fill the gap from the surrounding tokens.

The choice starts with the data you have, not with the type you like best. Ask whether you have explicit (input, output) pairs, whether you have plentiful task-related text without annotations, or whether your task is to change a part of a text while keeping the rest. The flowchart below follows that reasoning.

The flowchart is a starting point, not a rule that forbids combining stages. The source's own example chains them: a self-supervised stage on a large body of Vietnamese text can come before the task-specific stage. When cheap task-related data is plentiful and annotated data is scarce, it is reasonable to begin with self-supervised finetuning and reserve annotated examples for the later supervised stage.