Evaluating a Fine-Tuned LLM

By the time the chapter reaches this point, two large pieces of work are already finished. The first is a model that has been pre-trained and loaded into memory. The second is a dataset of instructions that has been cleaned, formatted, and turned into batches. Instruction fine-tuning is the step that connects these two pieces. The model is trained further, this time on instruction examples, so that it learns to respond to the kind of input the dataset describes.

Fine-tuning changes a model's behavior, but a changed model is not automatically a better one. After training, you still have to find out how well the model performs on the task you cared about. That question is the subject of this article: how a fine-tuned model is evaluated once training has finished. Keep one idea in mind as you read. Evaluation is a process that produces a number from a collection of model outputs, and each step hands a slightly different collection to the next step. The final number only makes sense if you understand what each step did to that collection.

Instruction fine-tuning is organized as a three-stage process. Stage 1 prepares the dataset. Stage 2 fine-tunes the LLM. Stage 3 evaluates the LLM. The diagram below places the steps described in this lesson under the stage each one belongs to, in the order they run. The full layout in the chapter numbers nine steps in total, and the steps shown here are the ones the lesson names.

Notice where the fine-tuning work sits. Stage 1 and Stage 3 surround it. Stage 1 produces the batched data loaders that the training step will consume, and Stage 3 looks at the responses the tuned model produces afterward. Stage 2 is the only stage in which the model's weights are trained, and it contains three steps: loading the pretrained LLM, running instruction fine-tuning, and inspecting the modeling loss.

Fine-tuning does not begin from random weights. The starting model is the one loaded in the previous section, and fine-tuning continues training that model. The data side is the instruction dataset prepared earlier in the chapter. That dataset was processed at the start of the chapter, so the fine-tuning step does not repeat that preparation work. In other words, the pretrained LLM supplies the starting weights and the general capability already learned, while the instruction dataset supplies the examples that shape how the model should respond.

Fine-tuning also needs a way to compute loss and run training. The chapter already contains code for both, so the fine-tuning code imports two functions from chapter 5 instead of writing those routines again: calc_loss_loader, which handles the loss calculation, and train_model_simple, which runs the training procedure. Four inputs meet at the single fine-tuning step: the pretrained LLM, which is loaded and then trained further; the instruction dataset, which supplies the examples; and the two chapter 5 functions, which supply the loss calculation and the training procedure. The diagram below shows how these inputs feed that one step.

Suppose a run of the process produces a result you did not expect. The three-stage structure gives you a way to narrow the search. Ask first which stage the problem most likely belongs to, then look only at the steps inside that stage. If the data loaders are wrong, the cause lies in Stage 1. If the model is loaded incorrectly or the loss and training steps misbehave, the cause lies in Stage 2. If the responses look wrong after training finishes, the cause may lie in Stage 3, where responses are extracted and scored.

Evaluation has its own internal progression. Each step changes the collection it works on: from inputs, to raw responses, to scored responses, and finally to a single average. The diagram below traces that collection. Read it from left to right and notice what the collection holds after each step.