Fine-Tuning a Classifier with LoRA

Fine-tuning with LoRA does not start by changing the pretrained model. It starts by deciding which numbers are allowed to change. A pretrained GPT model already holds a large set of learned parameters. LoRA is designed so that most of those original values stay exactly as they were, while a small number of new values are learned on top of them.

This means the workflow has a clear first move. Before any LoRA layer exists, you want a measurable checkpoint: a moment where you can count how many parameters are trainable, freeze the original ones, and count again. Those counts are how you confirm that the setup did what you intended. This article follows that sequence step by step, so that when a count looks wrong you know where to look.

The lesson's trace does three things in order: it counts the trainable parameters, runs a loop that sets requires_grad to False on every parameter, and then counts again. The first count is your baseline. It describes the original model as it was loaded, before anything is frozen or replaced.

Notice what the counting step is looking for. It is checking the requires_grad flag on each parameter. A parameter with requires_grad set to True is one the training process is allowed to update. The count is therefore a question about permission, not about how large the model is.

The flag belongs to each individual parameter, not to the model as a whole. A model is simply a collection of parameters, and freezing is a per-parameter setting that you apply by walking through that collection. This is why the freeze is written as a loop over the model's parameters rather than as a single switch on the model object.

The code above is an illustration of the loop the lesson describes, written for clarity. The output of 0 follows from the logic of the freeze: every original parameter has been switched off, and no new trainable parameters have been created yet. The lesson's own trace is where you should verify this against real printed numbers.

The order of operations is deliberate. The freeze happens first, and the replacement of Linear layers with LinearWithLoRA layers happens second. The reason is that the new LoRA layers are meant to be trained, so they must not be frozen. If the order were reversed, the freezing loop would also reach the newly created LoRA parameters, and no parameter in the model would be left trainable. The fine-tuning step would then have nothing to learn from.

The sequence can be read as a short chain. Load the pretrained model. Count its trainable parameters as a baseline. Freeze the original weights and count again. Replace the Linear layers with LinearWithLoRA layers. Count once more. Train. The diagram below lays out this chain so you can see where each checkpoint sits.

A LinearWithLoRA layer is not a brand new kind of layer that replaces the original one in a single step. It is a wrapper with two parts. The first part is the original Linear layer, which has been set to nontrainable. The second part is a new LoRA layer, which is trainable and is the piece that fine-tuning will update. Together these two parts make up the upgraded layer.

The diagram shows the wrapper's structure: one outer LinearWithLoRA layer holding a frozen linear child and a trainable lora child. The lesson's architecture printout confirms this structure, with each upgraded entry showing a linear child that holds the original weights and a lora child that is a LoRALayer. The lesson's material does not spell out the exact arithmetic for combining the two outputs, so the important point here is structural: the original weights are fixed, and the lora child is where the learning happens.

Not every component in the model is replaced. After the swap, the printed architecture shows LinearWithLoRA entries in the attention output projection, the two linear layers of the feed-forward block, and the output head. Components such as the GELU activation, the dropout layers, and the layer norms appear without a lora child, because they were not replaced.

The lesson prints the model architecture after the replacement so that you can check the change by eye. This printout is the second place to look when something seems wrong, after the counts themselves. It answers a structural question: which modules were swapped, and which were left alone.

The replacement call takes two settings, rank and alpha. The source uses a rank of 16 and an alpha of 16, and describes both values as good default choices. The rank matters for the counts you will check. Raising the rank increases the number of trainable parameters, because the LoRA layers grow with it. This is why a count that is higher than you expected can come from a rank setting as well as from a freeze that did not happen.

Alpha is commonly set to half, double, or equal to the rank. In this lesson the value is equal to the rank, at 16 in both cases.

There are three checkpoints in the sequence, and each answers a different question. The first count, taken before the freeze, describes the original model. The second count, taken after the freeze, should show that the original parameters no longer contribute trainable weight. The third count, taken after the replacement, describes the new LoRA parameters. The second and third counts measure different things, and mixing them up is one of the most common ways to misread the output.

Because each count is a checkpoint, a mismatch tells you where the process diverged. If the count after the freeze is not what you expected, the problem is in the freezing loop or in something that ran before it. If the count after the replacement is not what you expected, the problem is in the replacement step, or in the freeze having been applied to the model after the swap.

Suppose the trainable count after the freeze is not what you predicted. Work through the checkpoints in order. First, confirm the baseline count was taken before the loop. Second, confirm the loop ran over every parameter, since a freeze applied to only part of the model leaves some original weights trainable. Third, confirm that no replacement ran before the freeze. Only after these checks should you look at the replacement step itself.