Finetuning Techniques
Finetuning is performed by both model developers, who post-train models before release, and application developers, who adapt existing models to their own tasks or datasets.
Why Finetuning Is Broader Than Labeled Training
Suppose you are an application developer who has been handed a pre-trained language model and a deadline. Your first instinct might be that improving the model means collecting a large set of carefully labeled examples and training on them. That instinct captures only part of the picture. The phase of training that follows pre-training is called finetuning, and the word covers a wider range of methods than most people expect.
What do you think happens?
Does finetuning only mean training on a large set of carefully labeled examples?
Reveal answer
No, any training that happens after pre-training counts as finetuning, and it can be carried out with self-supervision, supervision, or reinforcement learning.
Because the definition is so broad, finetuning takes several different forms. Labeled examples are only one of them. Self-supervised finetuning, for example, uses task-related text that has no annotations.
This article follows the decisions that surround finetuning in order. Someone has to decide who performs the adjustment, which model serves as the starting point, whether the cost of adjusting the model is justified compared with simpler options, and which finetuning technique fits the goal. Finetuning is often described as a single step, but in practice it is a set of decisions. The sections below trace one model from its origin to the point where an application developer chooses how to adapt it.
Two Groups Who Finetune
Finetuning is not reserved for one group. Model developers finetune as part of building a model. They post-train it with different finetuning techniques before it is released. Application developers finetune as part of building a product on top of a model that already exists. Both groups perform finetuning, but they do not start from the same place.
The two groups also work at different levels. A model developer may release several versions of one model, each finetuned to a different extent. This gives application developers a choice. Instead of adjusting a single fixed model themselves, they can pick the version whose behavior best matches what their application needs.
One source example makes this concrete. A family of Code Llama models was produced by applying different finetuning techniques, so that each member of the family was tuned for a somewhat different purpose. The family shows that finetuning is a method a model developer can use to create a range of models from a shared starting point.
Tracing a Model From Pre-Training to Adaptation
Follow one model through the process. Training starts with pre-training, which is usually done with self-supervision. In that stage the model learns from a large amount of unlabeled data. For language models, this data is typically just sequences of text that need no annotations. The model at this point is a pre-trained model.
Next, a model developer post-trains the model with finetuning techniques. The model may be released in more than one version, each refined to a different degree. An application developer then chooses a version and, if needed, finetunes it further on data for a specific task. The diagram below shows these three states in order.
At each stage the model is the same underlying system, but the amount of task-specific adjustment it has received is different. The choice of starting point determines how much of that adjustment remains for the application developer to do. Starting from a more refined model means less adaptation work is left over.
Choosing Finetuning or Prompting
The decision to finetune comes before any technique is chosen. Compared with prompt-based methods, finetuning requires significantly more resources. The cost is not only data and hardware but also ML talent. Because of this difference, finetuning is generally attempted only after extensive experiments with prompt-based methods have been tried.
The two approaches are not competitors. Finetuning and prompting are not mutually exclusive, and real-world problems often require both. A system can use prompting for some parts of the task and finetuning for others. So the question is rarely which approach to choose forever. It is which one to try first, and whether the prompt-based results justify the extra cost of finetuning.
Deciding Whether a Support Team Should Finetune
A support team wants a model to sort incoming customer messages into categories. They have a pre-trained model and want to know whether to finetune it or to keep improving prompts.
- 1Start with prompt-based experiments
The team writes and compares several prompts that describe the categories and includes example messages. This is the cheaper route and it comes first in the source's ordering.
- 2Check whether the results are good enough
If the prompt-based results meet the need, finetuning is unnecessary and the team keeps prompting.
- 3Weigh the cost if results fall short
If the prompts are not enough, the team considers the significantly higher cost of finetuning, which includes data, hardware, and ML talent.
- 4Decide on a combined approach
If finetuning is justified, the team can still use prompting for parts of the task where it works well, since the two approaches can be combined.
Finetuning is attempted only after prompt-based experiments have been tried extensively, and only when the extra cost is justified by the results.
Three Learning Paradigms Behind Finetuning
Finetuning is defined by timing rather than by one method. Any training that happens after pre-training is finetuning. Pre-training itself is usually done with self-supervision, where the model learns from unlabeled data. Everything that comes after it counts as finetuning, regardless of which learning paradigm is used.
Finetuning can use one of three learning paradigms:
- Self-supervision, which learns from data without annotations, as in self-supervised finetuning
- Supervision, which learns from labeled input and output pairs, as in supervised and infilling finetuning
- Reinforcement learning, which learns from preference comparisons, as in preference finetuning
Four Finetuning Routes Compared
The finetuning techniques are siblings. Each answers a different question about the model. Self-supervised finetuning asks how to adapt the model with cheap, unannotated text. Supervised finetuning asks how to shape its responses with annotated examples. Preference finetuning asks how to push its outputs toward what people prefer. Long-context finetuning asks how to make it accept longer inputs. Infilling sits under supervised finetuning as one particular way of training.
| Technique | Data it needs | What it changes or teaches |
|---|---|---|
| Self-supervised finetuning | Task-related text without annotations | Adapts the model to a domain using raw sequences |
| Supervised finetuning | High-quality (input, output) pairs | Refines responses toward human usage and preference |
| Infilling finetuning | Text with a gap and the text around it | Teaches the model to fill in the blank |
| Preference finetuning | (instruction, winning response, losing response) records | Pushes outputs toward responses people prefer |
| Long-context finetuning | Longer sequences of training data | Changes the architecture so longer inputs can be handled |
Each finetuning technique is distinguished by the data it uses or the part of the model it changes.
Mistakes Beginners Make With Finetuning
Common mistake Assuming finetuning always means supervised training on labeled examples
Why it failsAny training after pre-training counts as finetuning, and it can use self-supervision, supervision, or reinforcement learning.
Do this insteadName the learning paradigm and the data type before deciding what kind of finetuning fits the problem.
Common mistake Finetuning before trying prompt-based methods
Why it failsFinetuning requires significantly more data, hardware, and ML talent, so it is generally attempted only after extensive prompt-based experiments.
Do this insteadRun prompt-based experiments first and finetune only when the results justify the extra cost.
Common mistake Treating prompting and finetuning as a permanent either-or choice
Why it failsThe two approaches are not mutually exclusive, and real-world problems often use both.
Do this insteadDecide which to try first, then consider combining them for different parts of the task.
Common mistake Expecting self-supervised data to need annotations
Why it failsSelf-supervised finetuning uses task-related data without annotations, which is why it is cheaper.
Do this insteadUse raw domain text first, and reserve expensive annotated data for supervised finetuning.
Common mistake Treating a preference record as an input with one output
Why it failsPreference data holds one instruction and two responses, and it marks which response was preferred.
Do this insteadRead each record as (instruction, winning response, losing response) and keep both responses.
Common mistake Assuming long-context finetuning is just more of the same training
Why it failsIt typically modifies the architecture, such as the positional embeddings, and may degrade performance on shorter sequences.
Do this insteadTreat it as an architectural change with a tradeoff, and check behavior on shorter inputs afterward.
Practice Choosing a Finetuning Route
A team has a pre-trained model, a large collection of raw domain documents, and only a small set of annotated question and answer pairs. Their prompt-based experiments have not yet been tried extensively. Which step should they take first, and which finetuning route would they consider first if prompts fall short? Explain your reasoning using the ordering of decisions and the cost of each option.
Hints
- Think about the experimentation order: prompt-based methods come before finetuning.
- Consider which finetuning route uses raw, unannotated text and which uses annotated pairs.
- Remember that prompting and finetuning can be combined.
Key Takeaways
- Finetuning is any training that happens after pre-training, and it can use self-supervision, supervision, or reinforcement learning.
- Model developers post-train models before release, and application developers most often start from a post-trained model because less adaptation work is then required.
- Finetuning needs significantly more data, hardware, and ML talent than prompt-based methods, so prompt-based experiments generally come first, and the two approaches can be combined.
- Self-supervised finetuning uses unannotated task text, supervised finetuning uses (input, output) pairs, and preference finetuning uses (instruction, winning response, losing response) records.
- Long-context finetuning typically changes the architecture, such as positional embeddings, and is harder to do, with a possible cost on shorter sequences.