Evaluating a Fine-Tuned Model

Fine-tuning changes how a model behaves, but a changed model is not automatically a better one. Once training has finished, you still have to find out how well the model performs on the task you care about. This article follows that question from start to finish. It begins with a complete fine-tuning run rather than a definition, then moves through the training log and timing output, and ends with the process that turns a fine-tuned model's responses into one performance number.

The run in this article uses LoRA layers. These are added to an existing model, and the model is then trained with the training function from chapter 6. The run also prints how long training took. Your job is to read the run first, form an expectation about its timing, and then use the printed output to check that expectation.

The easiest way to understand a training script is to read it in the order it runs. The source walks through the run in exactly that order, and each stage prepares something the next stage depends on.

The optimizer, the learning rate, the weight decay, and the epoch count all relate to the training function in one direction. The optimizer and its settings describe how each update is made, and the epoch count describes how many passes the training function makes over the data. Reading the setup as a list of inputs passed into one training call makes it easier to see which setting controls which behavior.

The timing result depends on two costs that move in opposite directions. LoRA layers add computation during the forward pass, so on a small example such as the one in the listing, training with them can be slower than training without them. Backpropagation, however, becomes more costly as models get larger. For larger models, training with LoRA is typically faster than training without it, because the cost saved in the backward pass outweighs the extra cost in the forward pass.

Because of this, the timing result depends on the size of the model and not only on whether LoRA is used. A result can look like a contradiction if you have only seen the small example. The source does not give a specific model size at which the two costs cross over, so treat the crossover as a trend rather than a fixed threshold.

The printed output is a log of the training run. Each entry gives an epoch number, a step number, a training loss, and a validation loss. The step numbers advance in increments of 50, because the training function was asked to evaluate every 50 steps. Accuracy lines appear only at some of those points, not at every entry.

A useful habit is to compare the training loss and the validation loss at the same step rather than looking at either one alone. In this run, the training loss keeps getting smaller across the epochs, while the validation loss goes up and down from one entry to the next. A training loss that keeps falling while the validation loss does not follow it is worth noticing, because it tells you the two measurements are not telling the same story.

The source gives three timing figures that are easy to mix up. Two describe typical hardware: about 15 minutes on an M3 MacBook Air laptop, and less than half a minute on a V100 or A100 GPU. The third describes the specific run printed in the listing. Read them side by side and notice which ones state their hardware and which do not.

Evaluation is a process that produces a number from a collection of model outputs. Each step hands a slightly different collection to the next step. The collection starts as inputs, becomes raw responses, then scored responses, and finally a single average. Tracking what the evaluation holds after each step is the easiest way to understand the final number.

Evaluation starts with a test set, which is a set of inputs the model was not trained on. The fine-tuned model is run on each item, and its responses are collected. At this point nothing has been judged yet, so the collection holds raw text responses only.

The next step is scoring. Each response in the collection receives a score, so the collection changes from a list of text responses into a list of numbers attached to those responses. The source describes the scorer as another LLM, and gives one concrete setup: an 8-billion-parameter Llama model run through the Ollama application, which can score the responses on the test set automatically.

The scores are then averaged, and the average serves as the number that quantifies performance. The test set may contain many examples, but the reported performance is a single value. Any question about one particular example can be answered only by going back to the scored collection before the averaging took place.

An average summarizes the test set but hides the spread of individual scores. The two scores in the illustration below are invented for teaching purposes and are not from the source. They show how a middle-looking average can come from two very different outcomes.