Introduction to Low-Rank Adaptation (LoRA)

Low-Rank Adaptation, usually called LoRA, is a way of finetuning a large model by training a small number of extra parameters instead of every weight in the model. Before you can use it, you have to configure it. A common first impression is that this is like flipping a switch: turn LoRA on, and the model adapts. The more accurate picture is that setting up LoRA is closer to filling out a form with two fields that interact.

The first field asks which weight matrices in the model should receive a LoRA adaptation. The second field asks what rank each of those low-rank factorizations should have. Both answers shape how much the model can learn during finetuning and how many parameters you end up training. This article explains the reasoning behind both fields. The goal is not to memorize one recommended configuration, but to understand why the choices matter and how a practitioner narrows them down when hardware limits the number of trainable parameters.

LoRA does not change the original model as a single block. It is applied to individual weight matrices, and each matrix can be chosen or skipped independently. Picture one weight matrix $W$ in the model. In a LoRA setup, $W$ stays frozen. Instead of retraining it directly, the training process learns a change to it, and that change is represented by two smaller matrices. The output of the adapted layer is the frozen path plus the low-rank path, so the frozen weight still does its original job while the adapter contributes a learned adjustment.

Read the diagram as a state change. The frozen matrix $W$ never moves during finetuning. The two adapter matrices are the only parts that learn. Their shapes come from the factorization: one matrix is $d \times r$ and the other is $r \times k$, where $r$ is the rank. A small $r$ means fewer trainable numbers, and the adapter's two small matrices together hold far fewer parameters than the full $d \times k$ matrix they stand in for.

LoRA has been tried on other kinds of networks, including convolutional neural networks, but it has been used most heavily on transformer models. Within a transformer, the most common targets are the four weight matrices inside the attention modules. These are the query matrix, the key matrix, the value matrix, and the output projection matrix.

The reason matrix choice matters is the budget. In an ideal world you would attach LoRA to all four attention matrices and let training sort out the rest. In practice, hardware memory often sets a ceiling on how many trainable parameters you can afford. Once that ceiling is fixed, the question changes from what is possible to what is best within the budget: given a fixed number of trainable parameters, which matrices should receive LoRA in order to get the strongest performance?

Once you have chosen matrices, you still need to decide the rank of each factorization. The source calls rank a critical parameter that affects performance. Rank is the size of the bottleneck between the two adapter matrices: it sets how many dimensions the learned change passes through. In the GPT-3 175B experiments the source describes, the rank used for the LoRA factorization was 2, and that setting produced the strong results reported for the all-four-matrices configuration.

The two fields should be read together rather than one at a time. A large rank applied to few matrices and a small rank applied to many matrices both spend parameters, so the two dials trade against each other under a fixed budget. The diagram below shows how the two decisions feed into the same total.

The source describes how Hu and colleagues handled this trade-off when finetuning GPT-3 175B. They set their trainable parameter budget at 18M, which is about 0.01 percent of the model's total parameters. That budget was enough to apply LoRA to a selected set of matrices, and the choice of which set mattered for the results.

The specific comparison the source reports is this. Applying LoRA to all four attention matrices with rank equal to 2 gave the best performance on the WikiSQL and MultiNLI benchmarks. WikiSQL is a benchmark built around translating questions into database queries, and MultiNLI is a natural language inference benchmark. Because this result was strong, it is the reference point for this configuration.

Sometimes the budget only allows two attention matrices instead of four. The source gives a practical fallback for this case: if you can choose only two attention matrices, the query and value matrices generally yield the best results. This is why the two-matrix case lands on that pair. The comparison below sets out the decision in the same way you would reason about it on your own setup.