Applying LoRA to GPT Models

Fine-tuning with LoRA does not begin by changing the pretrained model. It begins by deciding which numbers are allowed to change. A pretrained GPT model already holds a large set of learned parameters. LoRA is designed so that most of those original values stay exactly as they were, while a small number of new values are learned on top of them.

This article follows that decision in order. You will see how the original parameters are frozen, why the freeze must happen before any LoRA layers exist, what a LinearWithLoRA layer is made of, and how to use trainable counts as checkpoints. Those checkpoints let you find the exact step where something went wrong.

The important detail is that the flag belongs to each individual parameter, not to the model as a whole. The model is simply a collection of parameters, and freezing is a per-parameter setting that you apply by walking through that collection. This is why the freezing step is a loop over model.parameters() rather than a single switch on the model object.

The result you want is a clear, measurable checkpoint. Before freezing, you count how many parameters are trainable. You freeze them. Then you count again. The difference between those two counts is the visible effect of the freeze.

The diagram shows the key split. The original parameters move from trainable to frozen. The LoRA parameters are not part of the original model, so they are created later and remain trainable. The freeze only affects the parameters that exist when the loop runs.

The lesson trace does three things in order. It counts the trainable parameters, runs a loop that sets requires_grad to False on every parameter, and counts again. The sketch below follows that structure. It is an illustrative reconstruction of the steps described in the lesson, not a verbatim copy of the trace.

Notice what the prediction is asking. The counting line looks for parameters whose flag is True. The loop changes each parameter's flag. Reading both lines together tells you what the trace is measuring at each point.

The order of the two steps is deliberate. The freeze has to come before the LinearWithLoRA replacement, because the new LoRA layers are meant to be trained and must not be frozen. The source describes the reasoning this way: freeze the original weights first, then add the small trainable pieces.

The two counts in the trace measure different things. The first count is about the original model. The count after the replacement is about the new LoRA parameters. Mixing up these two measurements is one of the most common ways to misread the output.

A LinearWithLoRA layer is not a brand new kind of layer that replaces the original one in a single step. It is a wrapper with two parts. The first part is the original Linear layer, which has been set to nontrainable. The second part is a new LoRA layer, which is trainable and is the piece that fine-tuning will update. Together, the two parts make the upgraded layer.

When you look at the printed model architecture after the replacement, each upgraded entry shows the same structure. Every LinearWithLoRA entry has a linear child that holds the original weights and a lora child that is a LoRALayer. This printout is the second place to check when something seems wrong.

The replacement call takes two settings, rank and alpha. The source uses a rank of 16 and an alpha of 16, and it describes both values as good default choices. The rank controls the size of the LoRA layers. Raising the rank increases the number of trainable parameters, because the LoRA layers grow with it.

Alpha is commonly set to half the rank, double the rank, or equal to the rank. Changing alpha is a different kind of choice from changing rank, and it is not the setting that most directly controls how many parameters are trainable.

Verifying a freeze means treating each count as a checkpoint. The first checkpoint is taken before the loop. The second is taken after the freeze and before the replacement. The third is taken after the replacement. If one checkpoint disagrees with what you expected, the step just before it is the place to look.

Suppose the second checkpoint shows parameters still trainable after the loop. The problem lies in the freeze step, since the loop did not set every original parameter to False. Suppose instead the second checkpoint is zero but the third shows no trainable parameters at all. The replacement then either happened before the freeze or the LoRA layers were frozen afterward. Comparing each observed count with its expected value narrows the search to one step.