Training Function train_classifier_simple from Chapter 6

The LoRA fine-tuning run calls the chapter 6 training function, with an AdamW optimizer, a learning rate of 5e-5, a weight decay of 0.1, and 5 epochs.

Programming

Starting From a Complete Run

This article does not begin with a definition. It begins with a complete fine-tuning run: a script that adds LoRA layers to an existing model, trains that model with the training function from Chapter 6, and prints how long the training took. Your job is to read the run first, form an expectation about its timing, and then use the printed output to check that expectation.

The source introduces this step as the exciting part of the process, where the model is fine-tuned with the training function from Chapter 6. It also gives rough timings for the same kind of work on different hardware: about 15 minutes on an M3 MacBook Air laptop, and less than half a minute on a V100 or A100 GPU. Keep those numbers in mind. They set the context for the timing result you will read later, and you will see that they are not directly comparable to the run in the listing unless the hardware is known.

Following the Setup in Execution Order

The code is easiest to understand if you read it in the order it runs. The script first imports the training function from Chapter 6 and records a start time. It then fixes the random seed with torch.manual_seed(123), so that the run is set up the same way each time. Next it creates an AdamW optimizer over the model's parameters, with a learning rate of 5e-5 and a weight decay of 0.1. Those two settings decide how the trainable parameters are updated during training. The training call itself comes after the setup, and a second time reading follows it.

python

import time
start_time = time.time()
torch.manual_seed(123)
optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5, weight_decay=0.1)
# train_classifier_simple is the Chapter 6 training function
# it is called here with the optimizer, the data loaders, and 5 epochs
end_time = time.time()

import time
start_time = time.time()
torch.manual_seed(123)
optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5, weight_decay=0.1)
# train_classifier_simple is the Chapter 6 training function
# it is called here with the optimizer, the data loaders, and 5 epochs
end_time = time.time()

Notes on the code

The start time is recorded before any training work begins
The seed makes the setup repeatable across runs
AdamW decides how each trainable parameter is updated, using lr=5e-5 and weight_decay=0.1
The training call sits between the optimizer setup and the end time

1
Import the training function from Chapter 6 and record the start time
2
Fix the random seed with torch.manual_seed(123) so the setup repeats
3
Create an AdamW optimizer over the model's parameters with learning rate 5e-5 and weight decay 0.1
4
Call the training function for 5 epochs, which runs the training loop and evaluation
5
Record the end time and print the elapsed minutes

Where the LoRA Layers Sit in the Model

LoRA layers are added to an existing model. The model is not built from scratch in this run; the LoRA layers are attached to it, and then the training function trains that model. This matters for reading the timing, because the extra layers change what the forward pass computes, even though the training function itself is the same one used in Chapter 6.

Which parts of the run belong to the existing model, which parts are the added LoRA layers, and which parts the training function and optimizer act on

What the Timer Actually Measures

The timer wraps the whole training call. The start time is taken before the call and the end time after it, so the reported minutes include everything the training function does in that span, including any evaluation it performs during training. The timer does not isolate the forward pass or the backward pass. A printed number is therefore a total for a block of work, not a breakdown of where that time went.

Forming an Expectation About Speed

Before you look at the timing explanation, make a prediction. Suppose you train this model once with LoRA layers and once without them, using the same training function. The LoRA layers add computation during the forward pass, which pushes the run toward being slower. But backpropagation becomes more costly as models get larger, and this is where LoRA can pay off. The result therefore depends on model size, not only on whether LoRA is used.

What do you think happens?

On this small example, which run do you expect to take longer: the run with LoRA layers or the run without them?

The run with LoRA layers, because the extra layers add computation in the forward pass
The run without LoRA layers, because LoRA always saves time
They take the same time, because the training function is identical

Reveal answer

Answer:

The run with LoRA layers is expected to take longer on this small example.

LoRA layers add computation during the forward pass, so training with them can be slower on a small model such as the one in the listing. The advantage appears on larger models, where the cost saved during backpropagation outweighs the extra forward cost.

Factor	Small model (this example)	Larger model
Extra forward-pass work from LoRA layers	Present, and proportionally large	Present, but small relative to the whole model
Backpropagation cost without LoRA	Modest	Grows and becomes costly
Net expectation for LoRA training	Can be slower	Typically faster

How model size changes the balance between added forward cost and saved backward cost

The Training Loop, Batch by Batch

Inside the training function, each batch goes through the same sequence. The forward pass computes outputs, the loss is computed from them, the backward pass computes gradients, and the optimizer step updates the trainable parameters. This sequence repeats for every batch, and the whole pass over the data repeats for each of the 5 epochs. Evaluation is also run at intervals during training, every 50 steps in this run.

The order of the forward pass, loss, backward pass, and optimizer step for a batch, repeated across batches and epochs

Reading the Log Against Expectation

The printed output is a log of the training run. Each entry gives an epoch number, a step number, a training loss, and a validation loss. Step numbers advance in increments of 50, because evaluation was requested every 50 steps. Accuracy lines appear only at some of those points, not at every entry. Reading the log means tracking how the two losses move relative to each other over the run.

Worked example

Comparing the two losses at the same step

You are reading a log where the training loss falls across the epochs but the validation loss rises and falls between entries. Decide whether the two measurements agree.

1
Pick one step
Compare the training loss and the validation loss at the same step number, not either one alone.
2
Track the training loss
Follow the training loss across entries. In this run it keeps getting smaller over the epochs.
3
Track the validation loss
Follow the validation loss across the same entries. In this run it moves up and down from one entry to the next instead of following the training loss.
4
Judge the agreement
A falling training loss with an unsteady validation loss means the two measurements are not telling the same story, and that is the pattern worth noticing.

Result

The log shows a divergence between training and validation behavior, which is the thing to explain before drawing any conclusion about the timing or the quality of the run.

Timing and log reading go together. If the printed minutes differ from your expectation, check the log for where the run spent its steps, and check the hardware before deciding that the method itself is the cause.

Mistakes When Comparing Timings

Common mistake
Comparing a laptop timing with a GPU timing as if they measured the same thing
What it looks like
Setting the roughly 15 minute laptop figure beside the under half minute GPU figure and concluding the method differs between them
Why it fails
Timing figures are only comparable when the hardware is known. The two figures describe different hardware.
Do this instead
State the hardware for every timing you report and compare only runs on matching hardware, or explain the hardware difference openly
Common mistake
Assuming the printed minutes are the time for the forward or backward pass alone
What it looks like
Reading the total minutes and attributing all of it to backpropagation
Why it fails
The timer wraps the whole training call, including evaluation, so the number is a total for that block.
Do this instead
Treat the printed total as the whole call and look at the log to see when evaluation happened
Common mistake
Expecting LoRA to be faster on every example
What it looks like
Predicting a speedup for the small listing example because LoRA is a efficiency method
Why it fails
LoRA layers add computation during the forward pass, so on a small example the run can be slower.
Do this instead
Predict the direction from model size: the speedup is expected on larger models, where backpropagation is more costly
Common mistake
Comparing runs with different evaluation schedules or scopes
What it looks like
Setting one run that evaluates every 50 steps against another that evaluates less often and treating the totals as equal work
Why it fails
Evaluation is inside the timed span, so differing evaluation changes the total without changing the training method
Do this instead
Match the evaluation schedule and the measured scope before comparing totals
Common mistake
Reading one loss alone and concluding the model improved or worsened

Practicing the Read-Predict-Check Routine

MEDIUM

A colleague runs the same LoRA training function on a laptop and on a GPU and reports that the GPU run is far faster. Before you accept the comparison, list the three things you would need to know about each run to decide whether the timing difference reflects the method or the setup.

Hints

Think about the hardware named in the source timing figures
Think about what the timer covers, including evaluation
Think about whether the two runs used the same settings, such as the number of epochs and the evaluation schedule

Your attempt (not saved)

Key Takeaways

The LoRA run calls the Chapter 6 training function after fixing the seed with torch.manual_seed(123) and creating an AdamW optimizer with learning rate 5e-5 and weight decay 0.1, for 5 epochs.
LoRA layers add computation during the forward pass, so training with them can be slower on a small example, while on larger models the saved backpropagation cost typically makes LoRA training faster.
The timer wraps the whole training call, including evaluation, so the printed minutes are a total and are only comparable when the hardware is known.
Read the log by comparing training loss and validation loss at the same step, since a falling training loss with an unsteady validation loss signals that the two are not telling the same story.

More concepts starting with T