Preparing Instruction Datasets

Suppose you hand a language model a request such as 'Convert 45 kilometers to meters.' You want the model to respond with the answer, not with some loosely related continuation of the text. Instruction fine-tuning is the process built for exactly this purpose. Its aim is to strengthen an LLM's ability to follow an instruction and produce the response that a person actually wants.

The goal is not just any output. The desired response is the specific answer a careful person would give to that instruction. Instruction fine-tuning is the work of making a model's outputs look more like these targeted answers. Compared with a model that only continues text, a fine-tuned model is pushed toward responding to the instruction itself.

The before side of the diagram describes the behavior instruction fine-tuning is meant to move away from. The after side describes the behavior it is meant to produce: a response that fulfills the instruction.

Fine-tuning needs examples of the behavior you want. Each example pairs an instruction with a response that fulfills it. The source offers three illustrative pairs that show the range of tasks such data can cover.

The process is organized into three stages, carried out as nine steps in order. Stage 1 prepares the dataset that will be used for training. Stage 2 sets up a model and fine-tunes it on that dataset. Stage 3 evaluates the fine-tuned model. Read the structure as one line of work split into three blocks: the first turns raw material into a usable dataset, the second changes the model using that dataset, and the third asks whether the change was worthwhile.

The first stage begins with the data. Its job is to obtain the instruction dataset, look at it, and arrange it so that a training process can consume it. Within this stage the work proceeds through three steps. Taken together, they download, format, batch, and load the instruction-response pairs.

The dataset used in this process contains 1,100 instruction-response pairs of the kind shown above. The source notes that this dataset was created for the course text itself, and that readers who want other publicly available instruction datasets can find alternatives in appendix B of the course text.

The second stage takes the prepared dataset and applies it to a model. Two actions define it. First, a pretrained LLM is loaded. Second, that model is instruction fine-tuned on the prepared dataset. Stage 2 cannot begin until stage 1 has produced the dataset, which is why the three stages are presented in order.

Step 4 loads the pretrained LLM and establishes the starting model. Step 5 instruction fine-tunes that model, adjusting it using the instruction-response pairs. Step 6 inspects the loss, which lets you check how training is progressing as the adjustment runs. Stage 2 is the point where the data from stage 1 actually changes the model's behavior.

Once the model has been fine-tuned, the question becomes whether it is actually better at following instructions. Stage 3 answers that question through three steps. Step 7 extracts responses: the fine-tuned model is run to produce its answers. Step 8 is qualitative evaluation, which means reading and judging those answers. Step 9 scores the responses, turning the judgment into a score.

The order inside stage 3 matters. Evaluation looks at responses the model has actually produced, so it depends on stage 2 having finished. Qualitative reading comes before scoring, so the numbers are attached to answers someone has already examined.

To see how the stages connect, follow a single instruction, 'Provide a synonym for bright,' through the process. The walkthrough below is an original illustration built from the same pattern as the source's examples. It is not a result the source reports.