Fine-Tuning to Follow Instructions

Imagine you have built a language model and loaded pretrained weights into its architecture. You now have a model that has learned a great deal about language in general, but it is not yet aimed at any particular job. Ask it about a text message and it has no built-in commitment to answering 'spam' or 'not spam'. Something has to point it at the work you care about.

That pointing step is fine-tuning: continuing to train the model on a dataset that represents the specific task. Throughout this article we will keep one running task in mind, classifying text messages as 'spam' or 'not spam'. At every step, ask two questions: what should the model output for a given message, and how would anyone know whether it got the answer right?

Building a working LLM application is easier to follow as a sequence of stages. The first stage constructs the model. The second produces a general pretrained model and later adapts it toward assistant-style use. The third stage is where a pretrained model is adapted into a classifier. This article concentrates on that third stage.

The third stage starts from a pretrained model whose weights have already been loaded. It adds a labeled dataset, which says which class each example belongs to, and fines-tunes the model on that data. The output of this stage is a classifier. The path that matters runs from loaded weights and a labeled dataset, through fine-tuning for classification, and out to the classifier.

Picture a small collection of text messages. A person has read each one and marked it as spam or not spam. Fine-tuning for classification means training the pretrained model on pairs of message and label. Each training example is a message paired with its class label, so the model sees what a spam message looks like and what a legitimate message looks like, side by side with the answer it should produce.

The source names two main ways of fine-tuning an LLM, usually labeled step 8 and step 9. They start from different kinds of data, and that difference is what decides which one applies. Fine-tuning for classification uses a dataset in which each example already carries a class label, such as 'spam' or 'not spam'. Fine-tuning to follow instructions uses an instruction dataset, in which the model is shown a request and the response it should give.

A useful test is to look at the dataset first. If every example carries a class label, you are doing classification fine-tuning and the model's job is to pick a label. If every example is a request paired with a desired response, you are doing instruction fine-tuning and the model's job is to answer. The same pretrained model could be pointed either way; the data is what sets the direction.

When the model processes a message, it produces an output for each position in that message. A classifier needs one decision, not one per position, so a single output position is chosen and mapped to the class labels. In the common arrangement this is the output at the last token position, which is read and turned into 'spam' or 'not spam'. The other positions are processed along the way but are not the ones whose output is used for the label.

The read output is mapped to the labels the classifier was trained on. The label 'spam' and the label 'not spam' are two possible destinations for that one value. This is why a classifier can only return labels it was trained on: the output has been tied to that fixed set of classes during fine-tuning.

Once fine-tuning finishes, the model is supposed to be a classifier, but that is a claim that needs checking. The source treats evaluating the accuracy of a fine-tuned LLM classifier as an important step in its own right. The question evaluation answers is simple: when the classifier is given messages with known labels, how often does its predicted label match the true label?

The key detail is that evaluation uses examples the model did not learn from. Checking the classifier on its own training messages would show whether it memorized them, not whether it handles new ones. So the evaluation set is held out: labeled examples kept aside from training and used only to measure performance.

In words, accuracy is the share of evaluation examples that the classifier labels correctly. A classifier that gets most spam messages and most legitimate messages right has high accuracy. One that often confuses the two groups has low accuracy.

After fine-tuning and evaluation, the classifier can be used on messages it has never seen. A new message is passed to the model, and the model returns one of the labels it was trained on. The source describes this as using a fine-tuned LLM to classify new data. Nothing about this step retrains the model; it only reads the message and reports a label.

Reasoning about the returned label means keeping two things in view. First, the label is only as reliable as the accuracy measured on held-out examples. Second, a label is a best judgment among the classes the model knows, so an unusual message still receives one of those labels, even when the model is less sure of it. The label does not come with a guarantee of correctness for that particular message.