Finetuning Pre-Trained Language Models
Knowledge distillation, also called model distillation, trains a small student model to mimic a larger teacher model.
Predicting What Happens to the Teacher
Imagine you have trained a very capable language model. It answers questions well, but it is large, and running it for every user request takes substantial computing resources. The model works, yet the cost of putting it into a product may be too high, or responses may arrive too slowly. Knowledge distillation is one response to this kind of problem. Instead of accepting the size of the capable model as fixed, you train a second, smaller model to behave like it.
What do you think happens?
During distillation, does the teacher model's own weights change while the student is trained?
Reveal answer
No, the teacher is left unmodified
The teacher is not being modified in this process. Knowledge moves in one direction, from the large model into the small one. The teacher supplies the behavior to be imitated, and only the student's training is adjusted.
What Knowledge Distillation Means
Knowledge distillation, also called model distillation, trains a small student model to mimic a larger teacher model.
The name is a useful clue. In distillation, the knowledge held by the large model is passed along to the small one, much as a liquid is separated from a mixture and collected in a new container. The large model, called the teacher, does the work of producing behavior that the student should reproduce. The small model, called the student, is trained so that its behavior moves toward the teacher's behavior. The student is the one that ends up deployed, because it is the smaller and faster of the two.
The teacher and student do not play symmetric roles. The teacher exists to supply the behavior to be imitated, and the student exists to absorb that behavior in a compact form. The student is defined by the goal of mimicry, not only by its small size.
The Hand-Off From Teacher to Student
The source describes the flow during distillation as a sequence of hand-offs. Inputs go to the teacher. The teacher's behavior on those inputs guides the student. The student's training is then adjusted so that it imitates the teacher more closely. Read the diagram below from left to right and notice which model is changed at the end of the chain.
Trace the state of each model as training proceeds. At the start, the teacher already holds its capability and the student has not yet absorbed any of it. Each input passes through the teacher, which produces behavior the student should match. The student's training is adjusted in response. By the end, the student is closer to the teacher on those inputs, while the teacher is exactly as it was. The changed node in the diagram is the student, and only the student.
Why a Smaller Student Is Worth Building
Traditionally, the purpose of model distillation is to produce smaller models for deployment. Deployment means putting a model into real use, where it serves requests and must run on available hardware. A big model can be resource-intensive to deploy, so a smaller model that does the same job at lower cost is attractive. The source notes that distillation can produce a smaller, faster student model that retains performance comparable to the teacher.
Deciding Whether a Student Is Justified
A team has a capable language model that answers user questions well. Running it for every request is too costly, and responses are slower than the product needs. Should the team build a distilled student?
- 1Identify the cost problem
The capable model works, but its size makes each request expensive to serve and slow to answer. The problem is deployment, not answer quality.
- 2Ask what the student must retain
The student does not need to match the teacher exactly. It needs to keep performance comparable enough for the product's purpose.
- 3Assign the roles
The capable model becomes the teacher, whose behavior on inputs is the target. A smaller model becomes the student, trained to imitate that behavior.
- 4Judge the result
If the student is faster and cheaper to run while still doing the job well enough, distillation has achieved its purpose. If not, the trade-off was too large.
Distillation is justified here because the goal is a smaller, faster model that keeps much of the teacher's useful behavior, which is exactly what the cost problem requires.
Two Ways to Start a Student
The student does not have to start from nothing. The source identifies two ways it can be built. In the first, the student is trained from scratch. In the second, the student is finetuned from a pre-trained model. Both are still distillation, because in both cases the student's training is shaped by a larger model's behavior. What changes is where the student's training begins.
Following the Alpaca Pipeline
The stages behind Alpaca can be read in order. A large teacher produces examples, those examples are used as training data, and a pre-trained starting checkpoint is finetuned on them. Each stage consumes the output of the one before it, and the final stage yields the smaller model that is deployed.
Reading DistilBERT and Alpaca Results
The two reported results look similar in spirit, but they come from different setups. Reading them correctly means asking, for each claim, which part comes from the distillation itself and which part comes from the starting checkpoint or the teacher.
| Aspect | DistilBERT | Alpaca |
|---|---|---|
| Starting point | Trained from scratch | Finetuned from Llama7B, a pre-trained model |
| Teacher | BERT | text-davinci-003, with 175 billion parameters |
| Reported size | 40% smaller than BERT | 4% the size of the teacher |
| Reported behavior | Retains 97% of BERT's language comprehension | Behaves similarly to text-davinci-003 |
| Reported speed | 60% faster than BERT | Not reported in the source |
Reported results for the two students, as given in the source
Common Misreadings of Distillation
Common mistake Believing that distillation changes the teacher model
What it looks likeAssuming that both models are retrained together, so the large model also gets smaller or better during the process.
Why it failsThe teacher is not being modified in this process. Knowledge moves in one direction, from the large model into the small one.
Do this insteadTreat the teacher as a fixed source of behavior and the student as the only model whose training is adjusted.
Common mistake Expecting the student to be identical to the teacher
What it looks likeReading that a student retains 97% of BERT's comprehension and concluding the two models give the same answers.
Why it failsThe source says the student is expected to be comparable to the teacher, not identical to it. Distillation trades some capability for efficiency.
Do this insteadInterpret retained performance as a stated proportion of the teacher's capability, not as equality.
Common mistake Crediting distillation alone for a student's behavior
What it looks likeAttributing all of Alpaca's similarity to text-davinci-003 to the distillation step.
Why it failsAlpaca's starting point was a pre-trained model, Llama7B. Its behavior reflects that checkpoint as well as the distillation signal.
Do this insteadSeparate the starting checkpoint's contribution from the teacher-driven training when judging a reported result.
Common mistake Thinking only students trained from scratch count as distillation
What it looks likeDeciding that a model finetuned from a pre-trained checkpoint is not distillation because it did not start from nothing.
Why it failsBoth are still distillation, because in both cases the student's training is shaped by a larger model's behavior. What changes is where training begins.
Do this insteadClassify a student by its starting point, and recognize distillation by the teacher-driven training signal.
Common mistake Assuming a smaller model is always better for every purpose, when the goal of distillation is a smaller model that stays useful for the task at hand
Practice Classifying Students
A team takes a pre-trained language model that was never distilled, then trains it on outputs produced by a much larger model. Classify the student: is it trained from scratch or finetuned from a pre-trained model? And is the arrangement still distillation? Explain which part of the description tells you the answer.
Hints
- Look at where the student's training begins. Does the description name a pre-trained starting checkpoint?
- Ask whether the training signal comes from a larger model's behavior, since that is what defines distillation.
- Remember that the starting point and the teacher-driven signal are separate questions.
Key Takeaways
- Knowledge distillation trains a small student model to mimic a larger teacher model. The teacher is left unmodified, and the student is the model usually deployed.
- A smaller, faster student is useful when the large model is expensive to deploy, and the student is expected to retain performance comparable to the teacher rather than identical to it.
- A student can be trained from scratch, as DistilBERT was, or finetuned from a pre-trained model, as Alpaca was. Both are still distillation, because the training is shaped by a larger model's behavior.
- When reading reported results, separate the contribution of distillation from the contribution of the starting checkpoint and the teacher, and check which comparison each number makes.