Low-Rank Adaptation (LoRA) Technique

When people first configure LoRA, a tempting rule of thumb appears: if a little adaptation helps, a larger adapter should help more. Under that belief, the safest move is to set the rank as high as the hardware allows. This article tests that belief against what the research reports. Configuring LoRA means making three kinds of decisions: which weight matrices receive an adapter, what rank each factorization uses, and how the scaling hyperparameters are set. After that, you still have to decide how the adapted model will be served.

Keep the wrong belief in mind as you read. The rest of this article replaces it with a more accurate picture of how rank, matrix choice, and serving strategy interact, and it traces what actually happens to the numbers at each step.

LoRA can be applied to each individual weight matrix in a model. Its efficiency therefore depends on two things: which matrices receive an adapter, and the model's architecture, because different architectures have different weight matrices. LoRA has been used in other architectures such as convolutional neural networks, but it has been used primarily with transformer models.

In transformers, LoRA is most commonly applied to the four weight matrices inside the attention modules: the query matrix $W_q$, the key matrix $W_k$, the value matrix $W_v$, and the output projection matrix $W_o$. The choice is usually applied uniformly, so applying LoRA to the query matrix means applying it to every query matrix in the model.

Adapting more matrices, including the feedforward matrices, can yield better results. Databricks found that the biggest performance boost they obtained came from applying LoRA to all feedforward layers. Fomenko et al. reported that feedforward-based LoRA can complement attention-based LoRA, but attention-based LoRA typically offers greater efficacy within memory constraints.

The core idea of LoRA is that the frozen base weight W stays unchanged, while two small trainable matrices A and B learn a low-rank update. During a forward pass, the layer output is the base output plus the adapter's contribution. The adapter's product BA has the same shape as W, so it can be added directly to it.

Here $h$ is the layer output, $x$ is the input, $W$ is the frozen base matrix, $B$ and $A$ are the trainable low-rank matrices, and $\frac{\alpha}{r}$ is the scaling factor. The scaling factor is what ties alpha and r together, and it is discussed further below.

Rank is not the only hyperparameter. The value alpha controls how much the adapter's product contributes when it is merged into the base weights. Because the scaling factor is alpha over r, the two hyperparameters should be configured together. In practice, the ratio of alpha to r is typically set somewhere between 1:8 and 8:1, but the best ratio varies by task.

A useful intuition from the source is that if r is small, you might want alpha to be larger, and if r is large, you might want alpha to be smaller. Because the optimal pair depends on the task, experimentation is needed to find the best (r, alpha) combination for your use case.

The same budget can be spent in several ways. In the GPT-3 175B finetuning setup described in the source, Hu et al. set a trainable parameter budget of 18M, which is 0.01 percent of the model's total parameters. Within that budget, the parameters could go to one matrix at rank 8, two matrices at rank 4, or all four attention matrices at rank 2. The table below shows the idea for a single generated matrix size, where each option costs the same.

The source reports that on its benchmarks the value and output matrices scored higher than the query and key matrices in the single-matrix case. It also reports that applying LoRA to all four matrices at rank 2 gave the best performance on WikiSQL and MultiNLI. If you can choose only two attention matrices, the query and value matrices generally yield the best results.

The source gives two reasons a large rank can disappoint. First, a higher r can lead to overfitting, which means the adapter learns the training data too closely and generalizes worse. Second, a larger adapter costs more memory and training effort without a matching gain. Studies have shown that a small r between 4 and 64 is usually sufficient for many use cases, and a smaller r means fewer LoRA parameters and a lower memory footprint.

The opposite case also appears in the source: a researcher found that r = 256 achieved the best performance on his tasks. The lesson is not that large ranks are bad. The best rank depends on the task and must be tested. Turning r down risks underfitting if the task needs more capacity, so the right setting is found by experiment.

Once a LoRA model is trained, there are two general ways to serve it. The first merges the LoRA matrices A and B into the original model to create a new matrix W' before serving. Because no extra computation happens during inference, no extra latency is added. The second keeps W, A, and B separate during serving. The combination of the adapter with W then happens during inference, which adds latency.

The first option is generally better if you have only one LoRA model to serve. The second is generally better for multi-LoRA serving, which means serving multiple LoRA models that share the same base model. Keeping adapters separate adds some latency, but optimization techniques can reduce the added latency.