Instruction Fine-tuning
The general process of creating an LLM has two stages: pretraining, which trains on a large, diverse dataset to build a broad understanding of language, and fine-tuning, which refines the pretrained model.
Why a General Model Needs One More Step
Imagine a model that has read a very large amount of varied text. It has absorbed a broad sense of how language works, but it was not built for any one job. Perhaps you want it to sort your inbox into spam and not spam, or to follow instructions and answer in a particular format. A model that only learned general language does not automatically do these specific things well. This article is about the step that bridges that gap, called fine-tuning, and about the stage that comes before it.
Hold one question in mind as you read: what would you change about a general-purpose model if you wanted it to do one specific job reliably? The sections below answer that question step by step, and the last section asks you to apply the answer yourself.
The Two Stages of Building an LLM
The general process of creating an LLM has two stages. The first is pretraining, and the second is fine-tuning. The word pre in pretraining points to the fact that it comes first. In the first stage, a model is trained on a large and diverse dataset so that it develops a broad understanding of language. The result is a pretrained model, which then serves as a foundation that can be refined further in the second stage.
The key idea to carry forward is continuity. Fine-tuning does not begin from nothing. It begins from a model that already has broad language knowledge, and then adapts that model using a more focused dataset.
What Pretraining Contributes
The source describes pretraining in terms of a training objective. The LLM is trained on large text datasets with the goal of predicting the next word in a text. Learning to predict the next word over a huge and varied body of text is what builds the broad understanding of language that the pretrained model carries forward.
The data in pretraining is large text, and the objective is next-word prediction. Nothing in that objective says sort this email or answer in this format. Those specific abilities are not what pretraining is designed to deliver directly, which is exactly the gap fine-tuning is meant to close.
What Fine-tuning Adds
Fine-tuning trains the pretrained model further, but on a narrower, labeled dataset that is specific to a particular task or domain. Where pretraining works with large text and a next-word objective, fine-tuning works with labeled data: examples that come with the information the model should learn to produce or recognize for a specific purpose.
Think of the change this way. Before fine-tuning, the model has general language ability. After fine-tuning on a focused dataset, the same model is more reliable at the specific job the dataset describes. Pretraining supplied the broad foundation; fine-tuning supplied the targeted adjustment.
Two Fine-tuning Categories
The source identifies two categories of fine-tuning as the most popular. They differ in what the labeled examples look like, and the difference determines what the model learns from them.
Reading the Labeled Dataset
The clearest way to see the difference is to look at a single training example from each category. The two formats below are illustrative, written to show the structure the source describes rather than a particular library's required format.
- The instruction example carries a desired answer the model should learn to produce.
- The classification example carries a class label the model should learn to predict for the text.
| Aspect | Instruction fine-tuning | Classification fine-tuning |
|---|---|---|
| Labeled example contains | Instruction and answer pair | Text and associated class label |
| What the model learns | To follow instructions and produce the desired answer | To assign a text to the correct category |
| Typical output | A written response | One of a set of class labels |
The two most popular fine-tuning categories differ in the labeled data they use.
Choosing the Right Category
Given a dataset, you can decide which category fits by asking what each labeled example contains. The flowchart below turns that question into two checks. Follow the arrows from the top, answering each question about your data.
Matching a dataset to a category
You have a labeled dataset where each row is a customer message and the label is either complaint or question. Which fine-tuning category fits, and why?
- 1Check the structure of one row
Each row is a piece of text paired with a category label.
- 2Ask the first question
The row is not an instruction paired with a desired answer, so the first branch does not apply.
- 3Ask the second question
The row is a text paired with a class label, so the second branch applies.
- 4Choose the category
The dataset matches classification fine-tuning, because the task is to assign each message to one of the labeled categories.
Classification fine-tuning fits this dataset.
Mistakes Learners Make
Common mistake Treating fine-tuning as a replacement for pretraining
Why it failsFine-tuning begins from the pretrained model. The pretrained model is the starting point, not something that gets thrown away.
Do this insteadDescribe fine-tuning as a second stage that refines a model which already has broad language knowledge.
Common mistake Assuming pretraining data and fine-tuning data are the same kind
Why it failsPretraining uses large text datasets with a next-word objective. Fine-tuning uses labeled data tied to a specific task or domain.
Do this insteadCompare the two stages by data type and objective, not just by size.
Common mistake Choosing a category by the topic of the text rather than the structure of the labeled examples
What it looks likeDeciding a dataset is classification because its texts are about email.
Why it failsThe categories are defined by what each labeled example contains: an instruction with an answer, or a text with a class label.
Do this insteadInspect one labeled example and check whether it is an instruction-answer pair or a text-label pair.
Common mistake Assuming a general language model already performs a specific job reliably without any further adaptation
Practice: Classify Two Datasets
Dataset A: each row contains a sentence from a product review and the label positive or negative. Dataset B: each row contains an instruction such as Translate this sentence into French, paired with the French translation the model should produce. For each dataset, state which fine-tuning category fits, and name the two parts of one labeled example that justify your answer.
Hints
- Look at what each row contains before thinking about the topic.
- Dataset A pairs a text with a label. Dataset B pairs an instruction with a desired answer.
- Use the flowchart: start with the question about instruction and answer pairs.
Key Takeaways
- The general process of creating an LLM has two stages: pretraining on a large, diverse text dataset with a next-word prediction objective, and fine-tuning, which refines the pretrained model.
- Pretraining builds a broad understanding of language; fine-tuning adapts that pretrained model using a narrower, labeled dataset for a specific task or domain.
- Instruction fine-tuning uses instruction and answer pairs, while classification fine-tuning uses texts and their associated class labels.
- To choose a category, inspect the structure of one labeled example: an instruction paired with an answer points to instruction fine-tuning, and a text paired with a class label points to classification fine-tuning.