Bias Mitigation in Language Models

Imagine a general-purpose model that scores well on many public benchmarks, yet keeps stumbling on the one job your product needs done. It might write a fluent answer that still breaks the JSON format your software expects. It might translate queries correctly for standard SQL but fail on the dialect your company's database uses. The model is not broken. It was simply not trained much on your particular situation.

This is the situation that finetuning addresses. In this article we follow what happens to a model when it is finetuned, then examine three uses the source material singles out: shaping output structure and task performance, reducing biases, and producing smaller models that behave like larger ones. Keep the first situation in mind, because each of those uses starts from a model that is good in general but not yet good at your specific need.

The key word is continues. Because the model starts from its existing knowledge, finetuning is an adjustment rather than a construction from nothing. You are not teaching the model language from scratch. You are nudging a model that already knows a great deal toward the behavior your data demonstrates.

The source identifies the primary reason for finetuning as this improvement in quality, and names two kinds of gain that follow from it. The first is task performance: a model that is weak on a specific task can become stronger on that task once it sees examples of it. The second is structure: finetuning is commonly used to make a model produce outputs that follow a specific structure, such as JSON or YAML formats.

Consider what changes when a model is trained on examples of the output you want. The source gives two concrete cases of a model that is close to useful but not quite there. In the first, a model converts text to standard SQL well but fails with a less common dialect. Finetuning that model on data written in the less common dialect is expected to help. In the second, a model handles standard SQL for common queries but struggles with queries specific to one customer. Finetuning on those customer-specific queries may help in the same way.

The diagram above shows the shape of the change. The base model's reply is fluent but unstructured, which is exactly the failure the opening example described. After finetuning on examples that demonstrate the required format, the same kind of request produces a reply that follows the structure your software expects. The model's underlying knowledge is the same starting point; what has moved is the pattern it reliably produces for this task.

The source describes bias mitigation as an especially interesting use of finetuning. The reasoning starts from a limitation of training data: if a base model has absorbed certain biases from the data it was trained on, it will tend to repeat them. Finetuning offers a remedy. By exposing the model to carefully curated data during finetuning, you can counteract those biases.

The mechanism is straightforward to state. A bias is a default association the model has learned, such as linking a particular role with a particular kind of name. Curated data works against that default by presenting the underrepresented pattern often enough that the default association weakens.

The source also reports research by Garimella et al. (2022). They found that finetuning BERT-like language models on text authored by women can reduce these models' gender biases, while finetuning them on texts by African authors can reduce racial biases. The lesson is that the choice of finetuning data is itself the lever. Different curated sources target different biases.

The source notes that you can finetune a big model to make it even better, but finetuning smaller models is much more common. This leads to model distillation, which the source describes as a common approach: finetune a small model to imitate the behavior of a larger model, using data generated by that large model.

The training signal comes from the large model. It produces outputs, those outputs become the training data, and the small model is finetuned on that data so that it imitates the large model's behavior. Because the approach carries the larger model's knowledge over into the smaller one, it is called distillation. The source notes that it is covered in a later chapter together with other data synthesis techniques, so this article only introduces the core flow.

The source gives the practical reasons smaller models are finetuned more often. Smaller models require less memory, which makes them easier to finetune. They are also cheaper and faster to run in production. These are the trade-offs that make a distilled or finetuned small model appealing even when a large model is available.

A smaller model finetuned on one task can outperform a much larger out-of-the-box model on that task. The source gives a concrete case. Grammarly found that its finetuned Flan-T5 models (Chung et al., 2022) outperformed a GPT-3 variant specialized in text editing across a wide range of writing assistant tasks, even though the Flan-T5 models were 60 times smaller. The finetuning process used only 82,000 (instruction, output) pairs, which the source notes is smaller than the data typically needed to train a text-editing model from scratch.