Preparing a Dataset for Instruction Fine-Tuning

Once a model has been instruction fine-tuned, the natural next question is whether it actually got better. This article is about the stage where that question gets answered. Before any training happens, the instruction data has to be ready, and after training, the fine-tuned model's responses have to be judged. The two tasks named in this article, preparing a dataset and organizing instruction data in training batches, both concern the instruction data used in fine-tuning. The judging step, automated evaluation, comes at the very end of the overall process.

The early approach to checking a fine-tuned model was to take a handful of examples from the test set, look at what the model wrote in response, and judge the quality by eye. In the material this article draws on, that approach was limited to three examples.

The problem is not only that three examples are few. Three answers cannot tell you how often the model fails, which kinds of questions it struggles with, or whether a change you made last week helped or hurt. The method also does not scale. Reading hundreds or thousands of responses by hand is slow, tiring, and hard to repeat consistently from one review to the next. A judgment that depends on how tired the reader is at the time is not a dependable way to compare two versions of a model.

The solution the source develops is to let another, larger LLM do the scoring. In automated evaluation, the fine-tuned model writes responses to the test set questions, and a second model reads each response and assigns it a score. The second model does not need to be trained for this job in the course of the evaluation. It is an already trained model that is used only to read responses and score them.

The second model is described as larger, and that choice matters for the design. The evaluator needs to be at least as capable as the thing it is judging. If the judge is weaker than the model it scores, its scores would not carry much meaning. This is why the source moves from a small local model to a larger one when it discusses the options later on.

Compare this with manual sampling. Manual sampling has a person in the loop for every response, which is why it stalls at a few examples. In the automated version, the person designs the test set and reads the resulting scores, while the evaluator reads the responses. The same flow can be repeated on a larger set of responses without the reader being the bottleneck.

Instruction fine-tuning is organized as a three-stage process. The first stage prepares the dataset. The second stage fine-tunes the LLM. The third stage evaluates it. Automated evaluation belongs to the third stage. The source lays out nine numbered steps for the full process and shows which stage each one falls under, but the individual preparation steps are outside the scope of this article.

The source states the purpose of the third stage directly. Once the fine-tuned model has produced its responses, the goal is to quantify its performance by scoring those responses for the test set. This means the evaluator is not an extra add-on bolted onto the end of the work. It is the last step in the same pipeline that began with preparing the data.

The source develops the evaluator in two forms. The first version uses an existing instruction-fine-tuned 8-billion-parameter Llama 3 model developed by Meta AI. The second option uses a larger proprietary model. Each form has a different place where the evaluator runs and a different path the data takes.

Preparing a dataset sits inside supervised instruction fine-tuning. Supervised instruction fine-tuning is a process that trains a model using instruction data, and before that training can happen, the data has to be ready. Organizing instruction data in training batches is a related but separate task. It concerns how instruction data is arranged for training. Both tasks concern the same instruction data, but they are not the same job.

The easiest way to see how the two tasks relate is to follow the instruction data as it moves through them. The diagram below traces its state at each point. It shows the order given in the source's illustration, with preparation before batch organization, and it does not imply any order beyond that.

Read the diagram carefully. The first transition moves the collection from a raw state into a prepared state, which is the readiness that dataset preparation establishes. The second transition takes that prepared data and arranges it for training in batches. Neither transition, as the source describes it, is a training step, and neither is the evaluation stage that comes last.