Adapter Methods for Parameter-Efficient Finetuning

A large pre-trained language model may contain billions of weights. If you want it to behave differently for a particular job, retraining every one of those weights is expensive in memory, computation, and time. Parameter-efficient methods take a different route. They leave most of the model alone and train a small number of new values that steer its behavior. LoRA, which stands for Low-Rank Adaptation, is one of the most widely used methods of this kind.

The question this article answers is concrete. If LoRA does not train the original weights, where does the learned change live, how is it stored in a shape that fits the model, and why does it not slow the model down when it runs? The only reliable way to answer this is to follow one weight matrix from start to finish, so that is where we begin.

LoRA is not applied to the whole network as one object. You apply it to individual weight matrices. Take one of them and call it W. Suppose W has n rows and m columns, so its dimension is n by m. Everything LoRA does from here on is a response to this one matrix. It will produce a second matrix of the same n by m size that can be added to W, but it will never change W itself during training.

During finetuning only the new matrices are trained, and the original W is kept intact. This is the reason the method saves memory and computation: the large matrix is never updated, and the optimizer only has to track the small matrices that LoRA creates.

Once W is chosen, LoRA picks a number r. This number sets the size of the two smaller matrices. Matrix A has dimension n by r, and matrix B has dimension r by m. In shape notation, $A \in \mathbb{R}^{n \times r}$ and $B \in \mathbb{R}^{r \times m}$.

Notice how the inner dimension r is shared. The number of columns of A matches the number of rows of B, so the two can be multiplied. The product of A and B has dimension n by m, the same as W. That match is exactly what makes it possible to combine them later. The product $AB$ has shape $(n \times r)(r \times m) = n \times m$.

After A and B exist, LoRA forms a new matrix W' by adding the product AB to the original W. The product is scaled by a hyperparameter alpha, which controls how strongly the learned contribution affects the result. The new matrix W' takes the place of W inside the model.

Read the formula as a shape contract. A and B are the only pieces LoRA creates. Their product reconstructs something with the footprint of W, but it is built from far fewer numbers whenever r is small compared with n and m. The scale factor divides by r, so alpha and r work together. Alpha is a setting you choose to decide how much the update should contribute. Because the rank appears in the scale, changing r also changes how large the contribution is, not just how many parameters are stored.

The forward pass can be read in two equivalent ways. One reading is that the input passes through the frozen W and also through the scaled B then A path, and the two outputs are summed. The other reading is that the input passes through the single merged matrix W'. Because matrix multiplication distributes over addition, these give the same result. This is why the adapted model can be treated as one matrix.

LoRA is built on an older idea called low-rank factorization, a dimensionality reduction technique. The principle is that a large matrix can be written as the product of two smaller matrices. Fewer numbers are stored, so both computation and memory needs drop. The number of columns in the first factor, which equals the number of rows in the second, is the rank of the factorization. A full-rank matrix is the uncompressed original, while the two smaller matrices form a low-rank approximation of it.

The savings are dramatic, but the trade is that factorization is lossy. The pair only approximates the original matrix. A higher rank preserves more of the original information, while a lower rank compresses harder and discards more. In LoRA, this means that choosing r is a decision about how much change the update can express, not only how many numbers are stored.

A natural objection is that a model with many parameters should need many parameters to change its behavior. Researchers have argued otherwise. Although large language models have enormous parameter counts, their intrinsic dimension, meaning the true complexity of what they have learned, can be much lower than the number of parameters suggests. Pre-training appears to push this intrinsic dimension down, and larger models tend to end up with lower intrinsic dimensions after pre-training.

If that is true, pre-training works like a compression step that prepares the model for later tasks. A well-trained model needs only a small adjustment to be steered toward a new job. This is why a small set of trainable LoRA parameters and a modest amount of finetuning data can be enough. The source reports that for GPT-3, LoRA reached comparable or better performance than full finetuning on several tasks while training about 4.7 million parameters, roughly 0.0027 percent of the full finetuning count.

Following one matrix from start to finish shows the whole mechanism. LoRA chooses a weight matrix W of size n by m and freezes it. It chooses a rank r and creates A of size n by r and B of size r by m. Their product AB has the same shape as W. The product is scaled by alpha over r and added to W, producing W', which replaces W in the model. Because the change is merged into one matrix, the structure and speed of the model stay the same. The reduction in trainable values comes from factorization, and it works for finetuning because pre-training lowers the intrinsic dimension of the model.