Choosing Rank and Alpha for LoRA
LoRA is designed so that most of those original values stay exactly as they were, while a small number of new values are learned on top of them.
Deciding What Is Allowed to Change
Fine-tuning with LoRA does not start by changing the pretrained model. It starts by deciding which numbers are allowed to change. A pretrained GPT model already holds a large set of learned parameters. LoRA is designed so that most of those original values stay exactly as they were, while a small number of new values are learned on top of them.
This article focuses on the first half of that setup: freezing the original parameters, wrapping the layers with LoRA pieces, and checking the result with counts. Rank and alpha are the two settings you pass when the LoRA layers are created, and they are covered at the end of the article, after you understand what the counts are measuring.
Freezing Is a Per-Parameter Flag
Freezing sets requires_grad to False on every original parameter, so none of the pretrained weights are updated during fine-tuning.
The important detail is where the flag lives. It belongs to each individual parameter, not to the model as a whole. A model is simply a collection of parameters, and freezing is a setting you apply to each one by walking through that collection. This is why the freezing step is a loop over model.parameters() rather than a single switch on the model object.
- The loop visits every parameter in the model, one at a time.
- Setting
requires_gradto False marks that single parameter as not trainable. - The counting line adds up only the parameters still marked as trainable.
What do you think happens?
Suppose the freeze loop runs over every parameter of the original model, and no LoRA layers have been added yet. How many parameters will still count as trainable after the loop?
Reveal answer
None. The loop sets requires_grad to False on every original parameter, so the trainable count drops to zero.
The freeze is applied to every parameter the loop reaches. At this point the model contains only the original parameters, so nothing is left trainable. The LoRA layers come later, in the replacement step.
Why the Freeze Must Come First
The freeze is applied before the LinearWithLoRA replacement. The reason is that the new LoRA layers are meant to be trained. If the order were reversed, the freeze loop would also reach the newly created LoRA parameters, and they would be frozen too. Once that happens, none of the parameters in the model would be left trainable, and the fine-tuning step would have nothing to learn from.
The sequence is therefore deliberate: freeze the original weights, then add the small trainable pieces. The first count you take is about the original model. The count taken after the replacement is about the new LoRA parameters. The two counts measure different things, and mixing them up is one of the most common ways to misread the output.
Key Takeaways
- Freezing sets
requires_gradto False on each original parameter, so the pretrained weights are not updated during fine-tuning. - The freeze runs before the LinearWithLoRA replacement; reversing the order would freeze the new LoRA parameters too.
- A LinearWithLoRA layer wraps a frozen original Linear layer and a trainable LoRA layer.
- Trainable counts taken before the freeze, after the freeze, and after the replacement are checkpoints that locate where a result diverges.
- Raising the rank increases the number of trainable LoRA parameters; the source uses rank 16 and alpha 16 as good defaults, and alpha is commonly set to half, double, or equal to the rank.