Fine-Tuning Large Language Models on Instruction Data

Imagine you have finished building a language model that has already gone through the earlier work of this course. It can produce fluent text, but it has not been shaped for any particular job. Now you want to fine-tune it on instruction data, and the training run starts to struggle. It takes far too long, or the machine can barely run it at all.

Fine-tuning a language model on instruction data is demanding work. The size of the model you choose is one of the first things that determines how demanding it will be. This article looks at the two practical options the source material offers when hardware becomes the limiting factor: choosing a smaller model, and moving the computation onto a GPU. Both options answer the same underlying question: how much computational work does a training run ask the hardware to do, and how quickly can that hardware do it?

The two GPT-2 models used in this chapter differ mainly in size. GPT-2 medium has 355 million parameters, and GPT-2 small has 124 million parameters. Parameters are the adjustable numbers inside the model, and every training step has to work with them. The source states plainly that using and training the larger GPT-2 medium model is more computationally intensive than using and training the smaller one. In other words, the larger model asks the hardware for more computation on every pass through the data.

This relationship makes model size the first lever to consider when hardware is struggling. If the larger model is too demanding for the available machine, the source suggests a direct response: switch to the smaller model. The switch is a deliberate trade between capacity and workload. It is not a bug fix, because the larger model is not broken. It simply asks more of the machine.

The model is selected through a single setting named CHOOSE_MODEL. The source shows the two values that matter here. The larger option is written as gpt2-medium (355M), and the smaller option is written as gpt2-small (124M). Changing that one line is the entire switch. Nothing else in the training logic has to be rewritten.

Switching models is not the only route. The source also gives reference run times for training each model on several devices, including CPUs and GPUs. The table records how long two epochs of training took for each combination of model and device. Read it row by row to see the effect of the model, and column by column to see the effect of the device.

The two effects can be combined. Within any single model, the CPU is the slowest device and the GPUs finish much sooner. So the fastest entry in the table is the smaller model on the A100 GPU, and the slowest is the larger model on the CPU. Notice that the table does not show a single winner for every situation: the right choice depends on which device you have available and how much time you can afford.

A GPU is the second lever in the source. The text says that running the code on a compatible GPU requires no code changes and can significantly speed up training. The same program runs, and the hardware underneath it does the work faster. You do not rewrite the training logic to benefit from a GPU, and you do not change the model choice to get the speed-up. The two levers are independent of one another.

The GPUs named in the source are the NVIDIA L4 and the NVIDIA A100. The results shown in the chapter were produced with the GPT-2 medium model trained on an A100. The source also points to a supplementary section in the code repository that lists several options for using cloud GPUs, which is useful if you do not have a suitable GPU locally.

The course builds an LLM in stages. The third stage is the one this article covers. In that stage, the course takes a pretrained LLM and fine-tunes it. Read that sentence as a sequence: the pretrained model comes first, and fine-tuning is applied to it afterward. The diagram below shows that order as a flow of data from the pretrained model, through the fine-tuning step, to a model that is ready for specific tasks.

Notice the two halves of the fine-tuning definition. The first half names the starting object, a pretrained LLM, and the action performed on it, which is adjustment. The second half names the purpose, which is specific tasks. A learner who keeps both halves in mind will be less likely to confuse fine-tuning with building a model from nothing.

The source names two example tasks for a fine-tuned model: answering queries and classifying texts. It explains why they matter: these are among the most common tasks in real-world applications and in research. Each task asks something different of the model. Answering a query asks the model to produce a response to a question. Classifying a text asks the model to assign an input to a category. Both are ways of following an instruction, which is the behavior the fine-tuning stage aims to build.