Model Distillation

Imagine you have a general-purpose model that scores well on many public benchmarks, yet it keeps stumbling on the one job your product needs done. It might write a fluent answer that still breaks the JSON format your software expects. It might translate queries correctly for standard SQL but fail on the dialect your company's database uses. The model is not broken. It was simply not trained much on your particular situation.

This is the situation that finetuning addresses. Finetuning is a process for improving a model's quality, both in its general capabilities and in its task-specific capabilities. Model distillation is one common application of finetuning, and it is the main subject of this article. The sections below trace what happens to a model when it is finetuned, and then follow the path that distillation data takes from the original model to the smaller one.

Finetuning takes a model that already exists and continues training it on data that reflects what you want. Because the model starts from its existing knowledge, you are adjusting it rather than building it from nothing. The data you choose is the lever: it defines the behavior the model moves toward.

The source names two kinds of gain that follow from this improvement in quality. The first is task performance. A model that is weak on a specific task can become stronger on that task once it sees examples of it. The second is structure. Finetuning is commonly used to make a model produce outputs that follow a specific structure, such as JSON or YAML formats.

The diagram above is an illustration rather than a measured result. Read it as a before and after comparison of one kind of output. Before finetuning, the model may answer in fluent prose or produce an answer that does not match the format your software needs. After it has been trained on examples in the required format, its outputs move toward the structure in those examples, so the software can use them directly.

Finetuning is most useful when the model you want to use was not sufficiently trained on your task. The source gives a concrete case: an out-of-the-box model might convert text to the standard SQL dialect well but fail with a less common dialect. Finetuning that model on data written in the less common dialect is expected to help. A related case is a model that handles standard SQL for common queries but struggles with queries specific to one customer. Finetuning on those customer-specific queries may help in the same way.

The source describes bias mitigation as an especially interesting use of finetuning. It starts from a limitation of training data. If a base model has absorbed certain biases from the data it was trained on, it will tend to repeat them. Finetuning offers a remedy: by exposing the model to carefully curated data during finetuning, you can counteract those biases.

The source also reports a research finding. Garimella et al. (2022) found that finetuning BERT-like language models on text authored by women can reduce these models' gender biases, while finetuning them on texts by African authors can reduce racial biases. The lesson is that the choice of finetuning data is itself the lever. Different curated sources target different biases.

The source notes that you can finetune a big model to make it even better, but finetuning smaller models is much more common. The reasons are practical. Smaller models require less memory, which makes them easier to finetune. They are also cheaper and faster to run in production.

The example shows why a smaller model is attractive. It can be cheaper and faster to run while still performing strongly on the task it was finetuned for. Model distillation builds on this same attraction, but it changes where the training examples come from.

The source explains that in the early days of foundation models, the strongest models were commercial and offered limited finetuning access. There were not many competitive models available for finetuning, so the option was hard to use in practice.

That picture has changed. The open-source community now offers high-quality models of all sizes, tailored for a wide variety of domains. The source concludes that this proliferation has made finetuning a lot more viable and attractive. Teams have more starting points to choose from, including models sized to their memory and latency budgets, and each starting point can be adapted with the techniques described above.

The name comes from the way the approach carries the larger model's knowledge over into the smaller one. The source describes it as a common approach: finetune a small model to imitate the behavior of a larger model, using data generated by that large model. The source also notes that distillation is covered in a later chapter alongside other data synthesis techniques.

The question of why someone would train a model to imitate another has a practical answer. The original model already behaves the way its builders want. Distillation tries to reproduce that behavior in a model that is cheaper or faster to use. The goal is performance comparable to the original, not necessarily better performance.

Most training stories begin with a dataset collected from the world. Model distillation begins somewhere different. One model already exists and behaves the way its builders want, and the goal is to train a second model to reproduce that behavior. The second model is trained on data that the first model generated, not on a fresh collection of observations. That single change, where the training data comes from, is what makes distillation a form of data synthesis.

Read the diagram from top to bottom. The original model is not what gets trained. It acts as the source of examples. The distilled model is the only one that changes during distillation, and every change it undergoes is driven by the data the original model produced. Because the lessons come from the original model's outputs, the distilled model is never described as learning from raw observations in this setting.

Distillation involves two models with different roles. Keeping those roles separate is the simplest way to avoid confusion when reading about distillation. The table below lines up the two models across the properties the source describes.

The purpose of the original model is to already be the behavior worth copying. The purpose of the distilled model is to be a practical stand-in: it aims for comparable performance at a lower cost in memory, speed, or running expense. Notice that the performance goal is matching, not beating, the original.

Data synthesis is the process of generating data to be used for training models. Distillation is one application of that idea. Its generated data is produced by a model rather than by a separate collection effort. The source connects this to a wider trend: more models are being trained with synthetic data, and more techniques are being developed to synthesize data.

The concept map shows distillation as one branch of data synthesis, specifically the branch where a model supplies the data. The source mentions that distillation is one of several reasons people turn to data synthesis and refers to five such reasons, but the excerpt does not list all five. This article therefore describes only the reason it explains in detail: producing a cheaper and/or faster model with performance comparable to the original.

The source then moves to traditional data synthesis techniques, but the material available only introduces that section with its heading and describes no methods. So the techniques themselves are not covered here. What can be confirmed is their place in the larger picture: traditional techniques are a separate body of methods for synthesizing data, and distillation is one specific way to put data synthesis into practice.