Cloud GPUs for Model Training
Larger models such as gpt2-medium (355M) are more computationally intensive than gpt2-small (124M), because they have more parameters to work with during training.
When the Machine Cannot Keep Up
Fine-tuning a language model on instruction data is demanding work. Before you start a run, one of the first things that determines how demanding it will be is the size of the model you choose. Sometimes the machine in front of you cannot finish the job in a reasonable time, or it struggles to run the training at all. When that happens, the chapter points to two practical options: choose a smaller model, or move the computation onto a GPU. Both options answer the same underlying question, which is how much computational work a training run asks the hardware to do, and how quickly that hardware can do it.
What do you think happens?
Two versions of the same GPT-2 model are available: gpt2-medium with 355 million parameters and gpt2-small with 124 million parameters. Which one asks the hardware for more computation on every pass through the data?
Reveal answer
gpt2-medium asks for more computation on every pass.
Parameters are the adjustable numbers inside the model, and every training step has to work with them. The larger model has more of them, so each pass through the data demands more computation from the hardware.
Why Parameter Count Sets the Workload
The two GPT-2 models used in this chapter differ mainly in size. GPT-2 medium has 355 million parameters and GPT-2 small has 124 million parameters. The chapter states plainly that using and training the larger GPT-2 medium model is more computationally intensive than using and training the smaller one. In other words, the larger model asks the hardware for more computation on every pass through the data.
Parameters are the adjustable numbers inside a model. Every training step has to work with them, so the number of parameters is a direct measure of how much work each step carries.
Read the diagram as a chain of cause and effect. The model's size sets how many parameters every step must handle, and that count sets how much computation the hardware performs. A bigger count produces a heavier workload on each pass, even though the training code itself does not look any different.
Switching the Model in One Setting
The model is selected through a single setting called CHOOSE_MODEL. The larger option is written as gpt2-medium (355M), and the smaller option is written as gpt2-small (124M). Changing that one line is the entire switch. It is a deliberate trade between model capacity and workload, not a bug fix. When the hardware is the limiting factor, the chapter suggests switching to the smaller model.
- The string value selects which GPT-2 model the training run uses.
- Only this one line changes between the heavy and light setups.
- The rest of the training program stays the same.
Worked Example: Trading Capacity for Workload
A run that is too slow on the current machine
A training run using gpt2-medium (355M) on the available machine cannot finish in a reasonable time. What are the practical options, and what does each one change?
- 1Identify the limiting factor
The problem is the hardware. The model is heavy relative to what the machine can do, so the first lever to consider is model size.
- 2Reduce the model workload
Set CHOOSE_MODEL to gpt2-small (124M). The smaller model has fewer parameters, so every pass asks the hardware for less computation.
- 3Consider a compatible GPU
If a compatible GPU is available, run the same code on it. The program is unchanged, but the hardware underneath does the work faster.
- 4Combine the levers if needed
The two options are independent. A smaller model on a GPU is a valid combination, and the chapter's table shows that combining them gives the fastest entry.
Switching the model reduces the workload per pass, and moving to a GPU speeds up the same code. Neither option requires rewriting the training logic.
Reading the Run-Time Table
The chapter gives reference run times for training each model on several devices, including CPUs and GPUs. The table records how long two epochs of training took for each combination of model and device. Read it row by row to see the effect of the model, and column by column to see the effect of the device. Keep the two directions separate: comparing across a row holds the model fixed, and comparing down a column holds the device fixed.
| Reading direction | What is held fixed | What it reveals |
|---|---|---|
| Along a row | The model | The effect of the device on the same model |
| Down a column | The device | The effect of the model on the same device |
| Within one device | The device | The smaller model finishes faster than the larger one |
| Within one model | The model | The CPU is the slowest device and the GPUs finish much sooner |
How to read the run-time table in each direction
Two patterns stand out. First, within any single device, the smaller model finishes faster than the larger one. Second, within any single model, the CPU is the slowest device and the GPUs finish much sooner. The two effects combine. The fastest entry in the table is the smaller model on the A100 GPU, and the slowest is the larger model on the CPU. To answer a device-comparison question, hold the model constant and scan across the row.
What a Compatible GPU Changes
A GPU is the second lever. The chapter says that running the code on a compatible GPU requires no code changes and can significantly speed up training. The same program runs, and the hardware underneath it does the work faster. You do not rewrite the training logic to benefit from a GPU, and you do not change the model choice to get the speed-up. The two levers are independent of one another.
Mistakes When Choosing Model and Device
Common mistake Assuming a GPU requires rewriting the training code
What it looks likeRewriting the training loop before trying a GPU run.
Why it failsThe chapter states that a compatible GPU requires no code changes.
Do this insteadRun the same code on the compatible GPU and observe the speed difference.
Common mistake Treating the model switch as a bug fix
What it looks likeBelieving that gpt2-medium (355M) is broken and must be replaced.
Why it failsThe larger model is not faulty. It simply asks for more computation, and switching to the smaller model is a deliberate trade between capacity and workload.
Do this insteadSwitch CHOOSE_MODEL to gpt2-small (124M) when hardware is the limit, and accept the smaller capacity.
Common mistake Reading the run-time table in only one direction
What it looks likeComparing two devices while the model changes between the rows being compared.
Why it failsIf the model changes as well, the difference cannot be attributed to the device alone.
Do this insteadCompare within a row to isolate the device, and within a column to isolate the model.
Common mistake Expecting the GPU to make the larger model as light as the smaller one
Checking Your Reading of the Table
A training run of gpt2-medium (355M) on a CPU is too slow. Without changing the training logic, what are two changes you could make, and which of them alters the model's workload? Then, using the run-time table, explain how you would find the fastest entry and the slowest entry.
Hints
- Consider the two independent levers described in the chapter: the model choice and the device.
- The GPU change requires no code changes, while switching CHOOSE_MODEL to gpt2-small (124M) reduces the workload per pass.
- The fastest entry is the smaller model on the A100 GPU, and the slowest is the larger model on the CPU. Verify this by reading the table row by row and column by column.
Key Takeaways
- A larger model such as gpt2-medium (355M) is more computationally intensive than gpt2-small (124M) because it has more parameters, and every training step has to work with them.
- Switching CHOOSE_MODEL to the smaller model is a single-line change that reduces the workload a training run places on the hardware, and is a deliberate trade between capacity and workload.
- The run-time table is read by row to compare models on the same device and by column to compare devices for the same model; within any device the smaller model is faster, and within any model the GPUs finish far sooner than the CPU.
- A compatible GPU can significantly speed up training with no code changes, and it is independent of the model choice, so the two levers can be used separately or together.