Instruction Dataset Preparation

Instruction fine-tuning is a three-stage process: preparing the dataset, fine-tuning the LLM, and evaluating the LLM.

Programming

Two Pieces Waiting to Be Joined

By the time this point in the chapter is reached, two large pieces of work are already finished. The first is a model that has been pre-trained and loaded into memory. The second is a dataset of instructions that has been cleaned, formatted, and turned into batches. Instruction fine-tuning is the step that connects these two pieces. The model is trained further, this time on instruction examples, so that it learns to respond to the kind of input the dataset describes.

This article looks at the moment just before that training step. The goal is to understand what has been prepared, what the fine-tuning step is allowed to rely on, and how to tell which part of the overall process you are looking at when something goes wrong.

What do you think happens?

Before reading on: if the model's responses look wrong after fine-tuning finishes, which of the three stages is the most likely place to look first?

Stage 1, the dataset preparation
Stage 2, the fine-tuning itself
Stage 3, where responses are extracted and scored

Reveal answer

Answer:

Stage 3, where responses are extracted and scored

The source states that if the responses look wrong after training finishes, the cause may be in Stage 3. Problems with the data loaders point to Stage 1, and problems with loading the model or with the loss and training steps point to Stage 2.

The Three Stages at a Glance

Instruction fine-tuning is organized as a three-stage process. Stage 1 prepares the dataset. Stage 2 fine-tunes the LLM. Stage 3 evaluates the LLM. The overall process has nine numbered steps, each grouped under the stage it belongs to.

What happens next after the dataset is prepared, and how do the three stages hand off to one another?

Read the pipeline from left to right. Stage 1 does not train anything. Its job is to produce the batched data loaders that the training step will consume. Stage 2 is the only stage in which the model's weights are trained. Stage 3 comes afterward and looks at the responses the tuned model produces.

What Fine-Tuning Relies On

Fine-tuning has two inputs from earlier work. The starting model is the one loaded in the previous section. Fine-tuning does not begin from random weights. It continues training a model that already exists.

The data side is the instruction dataset prepared earlier in the chapter. That dataset was processed at the start of the chapter, so the fine-tuning step does not repeat that preparation work. Fine-tuning simply consumes the examples it has already been given.

What does each input supply, and how do the two differ in what they add to the final model?

The two inputs differ in what they add. The pre-trained model contributes the existing knowledge that training continues from. The instruction dataset contributes the examples that shape how that model is trained further, so that it learns to respond to the kind of input the dataset describes.

Worked example

Tracing the inputs into Stage 2

Stage 2 contains three steps: loading the pretrained LLM, running instruction fine-tuning, and inspecting the modeling loss. Which input does each step depend on?

1
Loading the pretrained LLM
This step brings in the model that was pre-trained and loaded earlier. Fine-tuning starts from this model rather than from random weights.
2
Running instruction fine-tuning
This step trains the loaded model on the instruction examples, using the batched data loaders produced in Stage 1.
3
Inspecting the modeling loss
This step checks the loss computed during training. The loss calculation comes from the chapter 5 functions rather than new code written for this step.

Result

The pre-trained model is the starting point, the instruction dataset supplies the training data through the batches, and the chapter 5 functions supply the loss calculation and training procedure.

Reusing Chapter 5 Functions

Fine-tuning needs a way to compute loss and run training. The chapter already contains code for both. Instead of writing those routines again, the fine-tuning code imports them. The two functions are calc_loss_loader and train_model_simple, and both come from chapter 5.

Importing rather than rewriting has a practical consequence. Each routine is defined once, in one place. Any fix or improvement to the loss calculation or the training procedure is made in that single definition and then reaches the fine-tuning step automatically.

Locating a Failure in the Process

Suppose a run of the process produces a result you did not expect. The three-stage structure gives you a way to narrow the search. Ask first which stage the problem most likely belongs to, and only then look at the steps inside that stage.

If a failure occurs, which stage does it belong to, and which other stages might it affect?

Read the flowchart as a sequence of questions. If the data loaders are wrong, the cause lies in Stage 1. If the model is loaded incorrectly or the loss and training steps misbehave, the cause lies in Stage 2. If the responses look wrong after training finishes, the cause may be in Stage 3, where responses are extracted and scored. The flowchart shows the search order; the source does not state a fixed order of checks beyond this stage-by-stage narrowing.

Mistakes When Reading the Process

Common mistake
Treating fine-tuning as starting from scratch
What it looks like
Assuming the fine-tuning step begins with random weights and so must learn language from nothing.
Why it fails
Fine-tuning continues training the pre-trained LLM that has already been loaded.
Do this instead
Start from the model loaded in the previous section and treat Stage 2 as further training of it.
Common mistake
Repeating dataset preparation inside the fine-tuning step
What it looks like
Re-cleaning and re-batching the instruction data while writing the training code.
Why it fails
The dataset was processed at the start of the chapter, and Stage 1 already produces the batched data loaders.
Do this instead
Consume the existing data loaders in Stage 2 rather than preparing the data again.
Common mistake
Rewriting the loss and training routines
What it looks like
Copying the loss calculation and training loop into the fine-tuning file.
Why it fails
calc_loss_loader and train_model_simple already exist in chapter 5, and importing them keeps a single definition.
Do this instead
Import both functions from chapter 5.
Common mistake
Blaming the fine-tuning step for every failure
What it looks like
Changing the training code when the responses look wrong after evaluation.
Why it fails
A wrong response may originate in Stage 3, where responses are extracted and scored, or in the chapter 5 functions.
Do this instead
Map the symptom to a stage first, then narrow the search to the steps inside that stage.

Practice: Placing a Failure

EASY

A run of the process finishes, and the batched data loaders contain the wrong examples. Which of the three stages does this failure belong to, and which other stage's output could it affect?

Hints

Look at which stage produces the batched data loaders.
Think about which later stage consumes those loaders.

Your attempt (not saved)

MEDIUM

After training, the model's responses look wrong, but the data loaders are correct and the model loaded without error. Name the stage you would examine first, and explain why the loss and training functions from chapter 5 are still worth checking.