Finetuning Language Models

Think about how you learn about the world. You use sight, sound, touch, and reading, and you rarely rely on just one of them at a time. A language model, by contrast, sees only text. It can write fluently, summarize an argument, and answer questions, yet it cannot look at a photograph of a broken appliance or listen to a recording of an engine. The gap between those two situations is the starting point for this article: what happens when models are built to take in more than words, and what those broader models can be used for.

This shift is best understood as an extension of language models rather than a replacement for them. Developers are widening what models can accept as input so that they can work in settings where information arrives in many forms. Keep that motivation in mind, because most of the concepts that follow grow out of it.

Models that handle a single modality have a long history. Models that handle several modalities at once are the newer development. The term multimodal model is used for any model that works with more than one modality, and that includes models that accept text together with another kind of input.

Two families of models fall under one umbrella. Large language models process text. Large multimodal models, which are generative, extend the same idea to several modalities. This article uses the term foundation models to cover both families. Many people still call systems such as Gemini and GPT-4V LLMs, but they are better described as foundation models.

Recall how a language model generates output. It produces the next token from the tokens it has already seen, and it repeats this step to build a response. A multimodal model keeps that same loop but changes what the conditioning context contains. Instead of only text tokens, the context can include image tokens alongside text tokens, or tokens from whichever other modalities the model supports.

The core operation does not change. The model still predicts a next token. What differs is the evidence available when it makes that prediction. A visual token can influence the wording of an answer just as an earlier word can, which is why a multimodal model can describe a picture it was shown.

Multimodal models need large amounts of data, just as language models do. Self-supervision works for them too. In self-supervision, the training signal comes from the data itself rather than from human-written labels. One variant is called natural language supervision, and it was used to train OpenAI's CLIP.

The method avoids writing a label for every image. Instead, training uses (image, text) pairs that already appear together on the internet. The source reports a dataset of 400 million such pairs, which it describes as 400 times larger than ImageNet, and it was produced without the cost of manual labeling. The result was a model that could generalize to several image classification tasks without additional training for each one.

The other major change is in task scope. Earlier models were usually built for one job. A model trained for sentiment analysis could not perform translation, and a translation model could not do sentiment analysis. Foundation models, because of their scale and the way they are trained, can handle a broad range of tasks. The shift is from task-specific models to general-purpose ones.

A general-purpose model often performs reasonably well straight out of the box. An LLM, for instance, can carry out both sentiment analysis and translation without being built separately for either. A general-purpose model can also frequently be adjusted to do better on one particular task.

AI engineering teams use three common techniques to adapt a general-purpose model: prompt engineering, retrieval-augmented generation (RAG), and finetuning. Each changes the model's behavior in a different way, so it helps to examine them separately. The main contrast is where the adaptation lives. Prompt engineering and RAG work around the model by controlling its input. Finetuning works inside the model by training it.

Finetuning trains a model on task-specific examples. Training adjusts the model's weights, which are the parameters that determine how it responds to input. After finetuning, the same kind of model can behave in a more targeted way for the task it was trained on.

In the diagram, the weights are the part that changes. The overall model is still the same model, now adjusted. The output the retailer receives moves from generic copy toward copy that reflects the brand's voice, which was the gap identified earlier.

In this pipeline the expensive pretraining step is absent. The model is already built when the engineer begins. The engineer's work is selecting an adaptation method, such as prompt engineering, RAG, or finetuning, and wrapping the result in an application. This approach is cheaper and faster, because foundation models make it cheaper to develop AI applications and reduce time to market.

Several factors together make AI engineering grow quickly. The first is that foundation models are powerful not only because they do existing tasks better, but because they can do more tasks. Applications once considered impossible are now feasible, and applications nobody had planned are appearing. Writing is a clear example. Since AI can now write as well as humans, and sometimes better, it can automate or partially automate any task involving communication, such as writing emails, responding to customer requests, and explaining complex material.