Evaluating a Finetuned Model
Finetuning adapts a base model that has some, but not all, of the capabilities needed so that it performs well for a specific task.
Why a Capable Model Can Still Fall Short
Imagine you have a model that already knows something useful, but not everything you need. It can handle part of the work, yet it falls short on the particular job you care about. That gap is the starting point for finetuning. Finetuning begins with a base model that has some of the capabilities required, and it adapts that model so it performs well enough on one specific task.
This article follows one idea from start to finish: a base model is a useful but incomplete starting point, finetuning is the process that closes the remaining gap, and evaluating a finetuned model means checking how well it now performs on the task it was adapted for. As you read, keep asking which task each model is actually prepared to perform.
Tracing a Base Model Toward a Specific Task
Since the early days of deep learning, transfer learning has offered a way to handle tasks where training data is limited or expensive to collect. The strategy runs in two stages. First, you train a base model on tasks where plenty of data is available. Then you transfer what that model has learned to the target task, which is the task you actually care about but for which data is scarce.
Follow the state of the model through this process. It starts as a model trained on data-rich tasks. After training it becomes a base model with some of the capabilities the target task needs. Finetuning then adapts that base model, and the result is a finetuned model that performs well on the specific task. The diagram below traces these stages in order.
Notice which stage produces which model. The base model is the output of the first training stage. The finetuned model is the output of the adaptation stage that starts from the base model. The base model is therefore not the final product for the target task; it is the input to the step that makes the final product.
Transfer Learning as the Umbrella
Transfer learning is the broader idea of taking knowledge gained from one task and using it to speed up learning on a new, related task. Finetuning is one way of doing transfer learning.
The concept was first introduced by Bozinovski and Fulgosi in 1976. Finetuning is therefore a specific route within that larger idea, not a separate subject that stands next to it. If you understand transfer learning as the umbrella, finetuning is one method that lives underneath it.
What is carried over is knowledge. A base model trained on data-rich tasks can pass the knowledge it gained along to a related task. This matters most when the target task has limited or expensive data, because the model does not have to learn everything about that task from scratch. Finetuning then adapts the transferred starting point so it works well on the specific job.
Google's multilingual translation system shows transferred knowledge at work. The system is able to translate between a language pair that has no direct training examples. The knowledge it gained from other language pairs carries over, so the missing pair is still handled.
Base Model Versus Finetuned Model
The base model and the finetuned model are closely related, but they are prepared for different levels of usefulness. The base model has some capabilities, though not all of those the task requires. The finetuned model is the result of adapting that starting point so it performs well for the specific task. The diagram below contrasts the two states.
| Question | Base model | Finetuned model |
|---|---|---|
| Which task is it prepared for? | A broader starting point with some of the needed capabilities | One specific target task |
| How complete is it for the target task? | Has some capabilities but not all of those the task requires | Has been adapted so it performs well on that task |
| What role does it play? | Input to the adaptation step | Final result of adaptation for the specific task |
Base model versus finetuned model
Closing the Gap in a Hypothetical Project
Deciding whether a base model needs finetuning
A team has a base model that handles some of the capabilities their task requires, but it falls short on the specific job. They want to know what finetuning would do and how to tell whether it worked.
- 1Name the target task
Be specific about the job. The finetuned model will be judged on this task, not on general ability.
- 2Identify the gap
Check which required capabilities the base model already has and which it lacks. The base model's partial coverage is the starting point.
- 3Finetune on the target task
Adapt the base model so it performs well on the specific task. The base model's existing capabilities are carried over rather than rebuilt.
- 4Evaluate on the target task
Judge the finetuned model by how well it performs on the task it was adapted for. Comparing against the base model on that same task shows what finetuning changed.
The team knows the base model was prepared for only part of the job, that finetuning closed the gap for the specific task, and that the finetuned model should be evaluated on that task rather than on general ability.
The before-and-after view shows the change in capability on the target task. Before finetuning, the base model handles only part of that task. After finetuning, the adapted model performs well on it. The change is measured on the task, which is why evaluation has to target the task and not the model in general.
Mistakes About Finetuning
Common mistake Thinking finetuning replaces the base model's knowledge with entirely new knowledge
Why it failsFinetuning builds on what the base model already learned. The base model has some of the required capabilities, and finetuning adapts that starting point rather than discarding it.
Do this insteadDescribe finetuning as adaptation of an existing model that already has part of what the task needs.
Common mistake Treating transfer learning and finetuning as two unrelated subjects
Why it failsFinetuning is one way of doing transfer learning. Transfer learning is the broader umbrella idea.
Do this insteadPlace finetuning under transfer learning as one specific method.
Common mistake Assuming a base model is useless until it is finetuned
Why it failsA base model has some of the capabilities needed. The source describes it as a useful starting point that can handle part of the work.
Do this insteadDescribe the base model as partially capable, and finetuning as the step that closes the remaining gap.
Common mistake Judging a finetuned model by general ability alone
Why it failsThe finetuned model is prepared to perform well on a specific task. Its quality should be assessed on that task.
Do this insteadEvaluate the finetuned model on the target task it was adapted for.
Common mistake Assuming transfer applies to any unrelated task
Why it failsTransfer learning uses knowledge from one task to speed up learning on a new, related task.
Do this insteadCheck that the new task is related to the task where the knowledge was gained.
Checking Your Understanding of the Two States
A team starts with a base model that has some of the capabilities their target task needs. They finetune it on that task. Describe, in your own words, what the base model is prepared to do, what the finetuned model is prepared to do, and what knowledge was carried over from the base model rather than rebuilt.
Hints
- Think about the base model as the output of the first training stage on data-rich tasks.
- Ask whether the base model is complete for the target task or only partly prepared.
- Consider which knowledge the source says is passed along to the new, related task.
Key Takeaways
- A base model has some, but not all, of the capabilities a specific task needs, so it is a useful starting point rather than a finished product.
- Finetuning adapts a base model so it performs well on one specific task, and it is one way of doing transfer learning.
- Transfer learning uses knowledge from one task to speed up learning on a new, related task, which helps most when target-task data is limited or expensive.
- A finetuned model should be evaluated on the specific task it was adapted for, not on general ability.
- Finetuning builds on what the base model already learned rather than replacing it with entirely new knowledge.