Deciding When to Finetune Models

Imagine two models. One is very large and performs well across many public benchmarks. The other is much smaller and, out of the box, performs worse on general tests. You are building a product that only needs one narrow task. Which model should you use?

Finetuning means adjusting a pre-trained model so it performs better on a specific task or dataset. The model begins as a general-purpose system. Finetuning shifts its behavior toward the work you care about, and the difference shows up when you compare the model before and after that adjustment.

The model's weights are adjusted, but the request itself stays the same. The only change is what the model has been trained to produce for that kind of input. This is why finetuning is a tool for task-specific behavior and not only for general quality.

The primary reason to finetune is to improve quality on two levels: general capabilities and task-specific capabilities. A model that scores well on a broad range of benchmarks may still handle your particular work poorly, because the training data it saw did not cover your task well. Benchmark strength and fit to your task are different things.

The source gives a concrete pattern for deciding when finetuning helps. An out-of-the-box model may convert natural language into standard SQL well but fail on a less common SQL dialect. In that case, finetuning on data that contains that dialect should help. A related case is a model that handles standard SQL for common queries but often fails on queries specific to a customer. Finetuning on customer-specific queries may help there too.

Finetuning is also commonly used to get a model to produce outputs in specific structures, such as JSON or YAML. If your application expects a certain format every time, a model that drifts from that format is a problem even when its answers are otherwise reasonable. A correct answer in the wrong shape can break the software that consumes it.

The source describes bias mitigation as an especially interesting use of finetuning. The reasoning starts from a known problem: if a base model carries biases from its training data, it may repeat them in its outputs. The proposed remedy is to expose the model to carefully curated data during finetuning, which can counteract those biases.

Consider the example the source gives. If a model consistently assigns male-sounding names to CEOs, finetuning it on a dataset containing many female CEOs can mitigate that pattern. The curation matters. The goal is not just more data, but data chosen so that it pushes against the specific bias you have observed.

The source also reports research by Garimella and colleagues from 2022. They found that finetuning BERT-like language models on text written by women can reduce those models' gender biases, and that finetuning them on texts by African authors can reduce racial biases. These findings show that the choice of finetuning text can target particular kinds of bias.

It is possible to finetune a big model to make it even better, but finetuning smaller models is much more common. The reasons are practical. Smaller models require less memory, which makes them easier to finetune. They are also cheaper and faster to use in production. Each advantage affects a different stage of a project: training, serving, and running costs.

The comparison is not a claim that small models are better at everything. The large model may remain stronger across broad benchmarks. The small finetuned model earns its place when the job is narrow, the data for that job is available, and the operating constraints of memory, cost, and speed matter.

Distillation is a way to pass the useful behavior of a larger model to a smaller one. The common approach is to finetune a small model so that it imitates the behavior of a larger model, using data that the large model generated. Because this approach distills the larger model's knowledge into the smaller model, it is called distillation. Data synthesis techniques of this kind are covered elsewhere in the course.