Preparing Instruction Data for Batching

Up to this point, instruction fine-tuning has relied on a small file of instruction examples. This article asks what changes when the same workflow is pointed at a far larger body of instruction data. Before reading on, think about what could go wrong when a training run becomes much bigger. Consider three questions: how long will it take, how much memory will it need, and how will you know whether the result is any good? The rest of this article answers those questions in order.

The Alpaca dataset is an openly shared instruction dataset of 52,002 entries. It is roughly fifty times larger than the earlier instruction file, and its entries are generally longer. It is openly available online, and the source presents it as an option you can substitute for the smaller instruction file used earlier in the chapter. The workflow you already know stays the same while the data changes.

Instruction fine-tuning on the Alpaca dataset follows the same overall plan as the smaller exercise, organized into three stages. The first stage prepares the data. The second stage loads a pretrained model and trains it on the instruction examples. The third stage tests what the trained model learned. Each stage depends on the output of the one before it, so the order matters.

Notice what each stage produces. The first two stages end with a trained model. The third stage does not train anything. It takes the trained model, asks it for answers to inputs it has not seen during training, and examines those answers.

The model in the first two stages is trained on the training portion of the instruction dataset. The evaluation stage uses a different portion, the held-out test set, which the training process did not use. Keeping the two portions separate is what makes the evaluation informative. A model judged only on examples it has already learned from tells you very little about whether it can handle new instructions.

Evaluation has two parts. First, the model-generated response is extracted for each input in the test dataset, and the responses are collected so they can be analyzed by hand. Second, the model is evaluated to quantify the quality of those responses. The source describes this second step as comparing the model's responses with the correct responses in the test set. Together these steps move from raw output to a measured judgment.

A dataset with 52,002 entries takes much more work to process than a small file. The source gives two practical recommendations. The first concerns hardware: because the dataset is large and the entries are longer on average, training is best done on a GPU, which accelerates the fine-tuning process.

The second recommendation concerns what to adjust when memory runs out. Two settings control how much work the hardware must hold at one time. The batch_size setting controls how many examples are processed together. The allowed_max_length setting caps how long each input may be. If training fails with out-of-memory errors, the source suggests lowering batch_size from 8 to 4, then to 2, or even to 1. It also suggests lowering allowed_max_length from 1,024 to 512, or to 256, to help manage memory.

Scaling instruction fine-tuning to the Alpaca dataset keeps the same three-stage workflow and changes the practical concerns: time, memory, and how results are judged. Each stage hands its output to the next, and the evaluation stage tests the trained model on inputs held out from training.