Traditional Data Synthesis Techniques
The data it generates carries the behavior that the distilled model is meant to reproduce.
Training a Model by Imitating Another
Most training stories start with a dataset collected from the world. Model distillation starts somewhere different. In distillation, one model already exists and behaves the way its builders want, and the goal is to train a second model to reproduce that behavior. The second model is trained on data that the first model generated, not on a fresh collection of observations. That single change, where the training data comes from, is what makes distillation a form of data synthesis.
What do you think happens?
During distillation, which model's parameters are changed by training?
Reveal answer
The distilled model
The original model is not what gets trained. It acts as the source of examples, and the distilled model is the only one that changes during distillation. Every change it undergoes is driven by the data the original model produced.
What Model Distillation Means
Model distillation trains a distilled model to imitate the behavior of an original model, usually to make it cheaper and/or faster.
Two details in this definition matter. The first is the word imitate: the distilled model is trained to reproduce what the original model does. The second is the reason for doing so. The usual aim is a model that is cheaper to run, faster to run, or both, while still reaching performance comparable to the original. The goal is comparable performance, not necessarily better performance. A distilled model is not expected to beat the model it imitates.
Why would anyone train a model to imitate another model? The original model may already behave the way its builders want, but it may be expensive or slow to use. Imitation offers a route to a smaller or quicker model that keeps the behavior that matters, and distillation is the technique that carries that behavior across from one model to the other.
Tracing Where the Training Data Comes From
The mechanism of distillation is easiest to understand as a path that data follows. First, the original model exists. Second, it produces data. Third, that generated data becomes the training material for the distilled model. In this setting the distilled model is not described as learning from raw observations. Its lessons come from the original model's outputs.
Read the diagram from top to bottom. The original model appears in the middle of the path as a producer, not as a learner. The data it generates carries the behavior that the distilled model is meant to reproduce. The distilled model sits at the end of the path, and it is the only component whose parameters are updated.
Comparing the Original and Distilled Models
Distillation involves two models with different roles. Keeping those roles separate is the simplest way to avoid confusion when reading about distillation. The original model is the existing model whose behavior is the reference. The distilled model is the second model, trained to imitate that reference. Their purposes, costs or speeds, and performance goals differ in the ways summarized below.
| Property | Original model | Distilled model |
|---|---|---|
| Role | Existing model whose behavior is the reference | Second model trained to imitate the reference |
| Role in training | Produces the data; is not trained | Is trained on the generated data; is the only model that changes |
| Cost or speed | Baseline | Usually cheaper and/or faster |
| Performance goal | The behavior to reproduce | Performance comparable to the original, not necessarily better |
The two models in distillation, compared across the properties the source describes
Worked Example: One Input-Output Round
Generating one training pair for a distilled model
An original model already exists and behaves the way its builders want. A team wants a smaller, faster model that behaves similarly. Show how one round of distillation produces a training example. This example is illustrative and generated for teaching; the source describes the process only in general terms.
- 1Choose an input
Pick an input that the distilled model should learn to handle. The original model receives this input.
- 2Record the original model's output
The original model produces an output for that input. This output is generated data, not a fresh observation from the world.
- 3Form an input-output pair
Pair the input with the output the original model produced. This pair is one piece of the training set.
- 4Repeat across many inputs
Repeating this over many inputs builds the full training set. The distilled model is then trained on these pairs, so its learning signal comes from the original model's behavior.
A training set of input-output pairs, all produced by the original model, which the distilled model is trained on to imitate that behavior.
Where Distillation Fits in Data Synthesis
Data synthesis is the process of generating data to be used for training models. Distillation is one application of that idea. The generated data is produced by a model rather than by a separate collection effort. The source notes a wider trend: more models are being trained on synthetic data, and more techniques are being developed to synthesize data.
The source also mentions that distillation is one of several reasons people turn to data synthesis, and refers to five such reasons. The material available here does not list all five, so this article describes only the reason it explains in detail: producing a cheaper and/or faster model with performance comparable to the original.
Mistakes Beginners Make with Distillation
Common mistake Assuming the distilled model learns from fresh real-world observations
What it looks likeThinking a distilled model is trained on a newly collected dataset of raw observations.
Why it failsIn distillation the training data is generated by the original model, which is the defining feature that makes it a form of data synthesis.
Do this insteadTrace the data back to its source: the training material comes from the original model's outputs.
Common mistake Treating the original model as the model being trained
What it looks likeBelieving both models are updated during distillation.
Why it failsThe original model acts as the source of examples. The distilled model is the only one that changes.
Do this insteadKeep the roles separate: the original produces data, the distilled model learns from it.
Common mistake Expecting the distilled model to outperform the original
Why it failsThe goal is performance comparable to the original, not necessarily better performance.
Do this insteadJudge a distilled model against comparable performance, along with its gains in cost or speed.
Common mistake Assuming distillation is the only form of data synthesis
Why it failsDistillation is one application of data synthesis, and the source refers to other reasons and to traditional techniques as a separate body of methods.
Do this insteadPlace distillation inside the wider category of data synthesis.
What This Article Does Not Cover
Checking Your Understanding of Distillation
A team has a large original model and wants a model that is cheaper to run while keeping similar behavior. Explain, in your own words, where the distilled model's training data comes from, which model changes during training, and what performance goal the team should set for the distilled model.
Hints
- Start from where the training data is produced, not where it is stored.
- Think about which model is the source of examples and which model is trained.
- Check whether the goal is comparable performance or better performance.
Key Takeaways
- Model distillation trains a distilled model to imitate the behavior of an original model, usually to make it cheaper and/or faster.
- The distilled model's training data is generated by the original model, which is why distillation counts as a form of data synthesis.
- The original model produces data and is not trained; the distilled model is the only model that changes during distillation.
- The goal is performance comparable to the original model, not necessarily better performance.
- Distillation is one application within the wider practice of data synthesis, which is the process of generating data for training models.