Fine-Tuning and Evaluating LLMs
The loaded model and the fine-tuned model are the same model at two points in time.
Why Start From a Trained Model
Building a language model that can hold a useful conversation is a large undertaking. A common shortcut is to begin with a model that has already been trained, and then continue its training for a new purpose. This article follows that shortcut. It covers two linked steps: loading a pretrained LLM, and then fine-tuning it so that it follows human instructions.
The source material for this topic is brief. It states the goal directly: a pretrained LLM is loaded and then fine-tuned to follow human instructions. The rest of this article unpacks what each part of that goal means and what changes to the model at each step, so the workflow is clear before you run any of it.
Tracing a Model From Loaded to Fine-Tuned
Before reading on, make a prediction. Think about the model at the moment it is loaded, and compare it with the model at the moment fine-tuning finishes. Which of those two states already contains the earlier training, and which one has been shaped toward following instructions?
What do you think happens?
Which state already contains the earlier training, and which state is shaped toward following instructions?
Reveal answer
The loaded model is the pretrained model that already holds the earlier training, and the fine-tuned model is that same model after further training has shaped it toward following human instructions.
Loading brings in an already trained model. Fine-tuning continues training that model on additional data, so the earlier training is the starting state and the instruction-following shape is what fine-tuning adds. Fine-tuning reshapes the existing model's behavior; it does not begin from an untrained model.
Read the diagram from left to right. The loaded model and the fine-tuned model are the same model at two points in time. What changes is its behavior: fine-tuning reshapes it so that it responds to instructions a person gives it. What stays the same is the starting point. The earlier training is not discarded, because fine-tuning begins from the state the model already has.
What Loading a Pretrained LLM Means
Loading a pretrained LLM means bringing an already trained model in as the starting point for further training.
The word loading is easy to read too literally. Loading does not mean asking a model to answer questions and stopping there. It means the model is now the starting point for further training. Whatever knowledge and parameter values it already holds form the state from which fine-tuning begins.
Because loading is the first half of the workflow, it sets up everything that follows. The model you load is not a blank slate waiting for data. It is a model that has already been trained, and the next step continues that training rather than replacing it.
The Goal of Instruction Fine-Tuning
Fine-tuning continues training a pretrained model on additional data so that it follows human instructions.
The phrase to follow human instructions names the target of the training. A pretrained model may produce fluent text without being shaped toward responding to a request. Instruction fine-tuning aims to change that: after training, the model should respond to instructions that a person gives it.
Reusing and Updating the Loaded Model
It helps to separate what the new training run reuses from what it updates. The loaded pretrained model is reused: its existing knowledge and parameter values are the state from which fine-tuning starts. The fine-tuning run then updates that state by training on additional data, with the aim of making the model follow human instructions.
Tracing one fine-tuning run
A team loads a pretrained LLM and then fine-tunes it on additional instruction-style data. For each stage, decide what is reused and what is updated.
- 1Stage 1: loading
The pretrained model is brought in as the starting point. Its existing knowledge and parameter values are reused, not rebuilt from scratch.
- 2Stage 2: fine-tuning
Training continues on the additional data. The model's behavior is reshaped so that it responds to instructions a person gives it. This is the part that is updated.
- 3Stage 3: the result
The fine-tuned model still carries the earlier training, because fine-tuning begins from that state. Its behavior now also reflects the instruction-following goal.
The pretrained knowledge is reused as the starting state, and the instruction-following behavior is what the fine-tuning run updates.
Misconceptions About Starting From a Pretrained Model
Common mistake Thinking that loading a pretrained LLM only means asking it questions
Why it failsLoading means the model becomes the starting point for further training. Questions alone do not make it a starting point for training.
Do this insteadTreat loading as the step that brings a trained model in before fine-tuning continues its training.
Common mistake Assuming fine-tuning begins from an untrained model
Why it failsFine-tuning reshapes the existing model's behavior. It does not begin from an untrained model, so the earlier training is the state it starts from.
Do this insteadThink of fine-tuning as continued training on additional data, not as training from zero.
Common mistake Assuming that a fluent pretrained model already follows instructions
Why it failsA pretrained model may produce fluent text without being shaped toward responding to a request. Following instructions is the target that instruction fine-tuning aims for.
Do this insteadTreat fluency and instruction-following as different properties, and expect fine-tuning to be what shapes the second one.
Practice Tracing a Fine-Tuning Run
A colleague says: 'We loaded a pretrained LLM, so we are starting over and the model knows nothing yet. Fine-tuning will teach it everything from the beginning.' Using the workflow in this article, explain what is wrong with this statement. In your answer, say what state the model is in when it is loaded, and what fine-tuning changes.
Hints
- Look at the definition of loading: what does the loaded model become?
- Compare the phrase fine-tuning reshapes the existing model with the phrase starts over.
- Consider which state already contains the earlier training.
Key Takeaways
- Loading a pretrained LLM means bringing an already trained model in as the starting point for further training.
- Fine-tuning continues training that model on additional data so that it follows human instructions, and it reshapes the existing model's behavior rather than starting from an untrained model.
- The workflow is two linked steps: load the pretrained LLM, then fine-tune it to follow instructions. The loaded model and the fine-tuned model are the same model at two points in time.
- Common misconceptions include thinking loading only means asking questions, assuming fine-tuning starts from scratch, and assuming a fluent model already follows instructions.