Instruction Datasets for Fine-Tuning
Instruction fine-tuning adapts a pretrained LLM to follow human instructions and generate desired responses.
A Trained Model That Still Needs Direction
A pretrained LLM has already been trained, yet training alone does not tell it how to respond to a person who gives it an instruction. A model that has only been pretrained has learned from its training process, but that learning does not, by itself, teach it to answer a request the way a person wants. This gap is what the topic of this article addresses.
What do you think happens?
A pretrained LLM has already completed training. Does that mean it knows how to respond when a person gives it an instruction?
Reveal answer
No, training alone does not tell it how to respond to a person who gives an instruction.
Pretraining gives the model its learned starting point, but the step that adapts it to follow human instructions and produce the responses people want is instruction fine-tuning.
What Instruction Fine-Tuning Adapts
Instruction fine-tuning adapts a pretrained LLM to follow human instructions and generate desired responses.
The key word in this definition is adapting. The model is not built from scratch. It starts from a pretrained LLM and is adjusted so that its behavior shifts toward following instructions. The definition names two outcomes: the model follows human instructions, and it generates the responses that were desired.
The goal of this article is to understand the process itself: what goes into it, what happens during it, and what the model looks like afterward. Each of those questions is answered by following the model through the process.
Following the Model From Start to Finish
The easiest way to picture instruction fine-tuning is to follow the model from its starting state to its finished state. The model begins as a pretrained LLM. It then passes through a training loop that is fed by an instruction dataset. It ends as a fine-tuned LLM, which is the model that follows human instructions.
Read the diagram from left to right. The pretrained LLM enters the process as the model being adjusted. The output is the fine-tuned LLM, the only node marked as changed, because it is the one whose behavior has been shifted toward following human instructions. The pretrained model and the fine-tuned model are the same kind of object at different points in the process; the difference is the state each one is in.
The Three Ingredients
Instruction fine-tuning combines three ingredients. Each one plays a distinct role, and the process cannot be understood without all three. The pretrained model is the object being adjusted. The instruction dataset supplies the examples that drive the adjustment. The training loop is the procedure that runs the model through training iterations using those examples.
- Pretrained model: the LLM that has already been trained and is the starting point for adaptation
- Instruction dataset: the instruction data that the model continues training on, with the aim of changing its behavior toward following instructions
- Training loop: the procedure by which the model is run through training iterations on the instruction data
Comparing the Pretraining and Fine-Tuning Loops
The pretrained model is fine-tuned on the instruction dataset using a training loop similar to the one used in pretraining. The mechanics of the learning procedure are therefore familiar: the model is run through training iterations, just as it was during pretraining. What differs is the data. Pretraining was the first training stage. Instruction fine-tuning continues training, but on instruction data, with the aim of changing the model's behavior toward following instructions.
| Aspect | Pretraining | Instruction fine-tuning |
|---|---|---|
| Position in the process | The first training stage | Continues training after pretraining |
| Training loop | A training loop that runs training iterations | A training loop similar to the one used in pretraining |
| Data | Pretraining data | Instruction dataset |
| Aim | Produces the pretrained model | Shifts behavior toward following human instructions and producing desired responses |
What stays the same and what differs between the two training stages, as described in the source
Tracing the Model's State Step by Step
The following walkthrough tracks one model across the process. It is a generated illustration of the state sequence described in the source, not a record of a real training run.
Following one pretrained model through instruction fine-tuning
A pretrained LLM continues any text it is given but has not been adapted to follow instructions. What state does it hold at each stage of instruction fine-tuning, and what is its final state?
- 1Starting state
The model is a pretrained LLM. It has already been trained, but training alone does not tell it how to respond to a person who gives it an instruction.
- 2Entering the process
The pretrained model enters as the model being adjusted. The instruction dataset supplies the examples that drive the adjustment.
- 3Passing through the training loop
The model is run through training iterations, similar to pretraining, but using instruction data. Its behavior is shifted toward following instructions.
- 4Final state
The model is now a fine-tuned LLM. It is the changed state at the end of the process and the model that follows human instructions.
The same kind of object moves from a pretrained LLM to a fine-tuned LLM. The change is in its state: it now follows human instructions and generates the responses people want.
Common Misreadings of Fine-Tuning
Common mistake Thinking instruction fine-tuning builds the model from scratch
Why it failsThe definition says the model starts from a pretrained LLM and is adjusted. The key word is adapting.
Do this insteadTreat instruction fine-tuning as adjusting an existing pretrained model, not as creating a new one.
Common mistake Assuming a pretrained model already follows instructions
Why it failsTraining alone does not tell a pretrained LLM how to respond to a person who gives it an instruction.
Do this insteadRecognize that the adaptation step is what makes the model follow human instructions.
Common mistake Believing the training loop is a completely different procedure from pretraining
Why it failsThe source says fine-tuning uses a training loop similar to the one used in pretraining.
Do this insteadFocus on the change in data and purpose rather than assuming a new learning mechanism.
Common mistake Confusing the pretrained model and the fine-tuned model as different kinds of object
Checking Your Understanding of the State Change
Without looking back, describe the path of one pretrained LLM through instruction fine-tuning. Name the three ingredients, say which one is the changed state at the end, and explain in one sentence what differs between the pretraining loop and the fine-tuning loop.
Hints
- Start with the pretrained model, then ask which ingredient supplies the examples.
- The changed node at the end of the diagram is the fine-tuned LLM.
- The source says the loop is similar to pretraining; think about what changes in the data.
Key Takeaways
- Instruction fine-tuning adapts a pretrained LLM to follow human instructions and generate desired responses, starting from the existing model rather than building one from scratch.
- The process combines three ingredients: the pretrained model, the instruction dataset, and the training loop.
- The model moves from a pretrained LLM to a fine-tuned LLM, which is the changed state at the end of the process and the one that follows human instructions.
- The fine-tuning training loop is similar to the pretraining loop; the main difference named in the source is that fine-tuning uses instruction data to shift behavior toward following instructions.