Intrinsic Dimension of Pre-Trained Models

A large pre-trained language model may contain billions of weights. If you want it to behave differently for a particular job, retraining every one of those weights is expensive in memory, computation, and time. Parameter-efficient methods take a different route. They leave most of the model alone and train a small number of new values that steer its behavior. LoRA, which stands for Low-Rank Adaptation, is one of the most widely used methods of this kind.

The question this article answers is concrete. If LoRA does not train the original weights, where does the learned change live, how is it stored in a shape that fits the model, and why does this not slow the model down when it runs? The best way to answer this is to follow one weight matrix from start to finish, so we will begin there.

LoRA is not applied to the whole network as one object. You apply it to individual weight matrices. Take one of them and call it W. Suppose W has n rows and m columns, so its dimension is n by m. Everything LoRA does from here on is a response to this one matrix. It will produce a second matrix of the same n by m size that can be added to W, but it never changes W itself during training.

LoRA begins by choosing a number r, which sets the size of the two smaller matrices. Matrix A has dimension n by r, and matrix B has dimension r by m. The inner dimension r is shared. The number of columns of A matches the number of rows of B, so the two can be multiplied. The product of A and B has dimension n by m, the same as W, and that match is exactly what makes it possible to combine them later.

Once A and B exist, LoRA forms a new matrix W' by adding the product AB to the original W. The product is scaled by a hyperparameter alpha, which controls how strongly the learned contribution affects the result. The new matrix W' takes the place of W inside the model.

The division by r inside the scaling term means that alpha and r work together. The rank enters the scale, so changing r changes how large the contribution is, not only how many parameters are stored. Keep this in mind when you change the rank, because the strength of the update moves with it.

During a forward pass, the input takes two paths and the results are summed. One path goes through the frozen W. The other goes through the small factors A and B, is scaled, and is added to the first result. Because the sum is folded into a single matrix, the model keeps its original layer structure.

LoRA builds on an older idea called low-rank factorization, a dimensionality reduction technique. The principle is that a large matrix can be written as the product of two smaller matrices. Fewer numbers are stored, so both computation and memory needs drop. The number of columns in the first factor, which equals the number of rows in the second, is the rank of the factorization. A full-rank matrix is the uncompressed original, while the two smaller matrices form a low-rank approximation of it.

In LoRA the same counting applies to the update. A full n by m update has $n \times m$ entries. The two factors A and B together have $r(n + m)$ entries. When r is small compared with n and m, the factored form is much smaller. The gap widens as the matrices grow, but it also widens as r grows, because each extra unit of rank adds $n + m$ new numbers.

A natural objection is that a model with many parameters should need many parameters to change its behavior. Researchers have argued otherwise. Although large language models have enormous parameter counts, their intrinsic dimension, meaning the true complexity of what they have learned, can be much lower than the number of parameters suggests. Pre-training appears to push this intrinsic dimension down, and larger models tend to end up with lower intrinsic dimensions after pre-training.

If that is true, pre-training works like a compression step that prepares the model for later tasks. A well-trained model needs only a small adjustment to be steered toward a new job, which is why a small set of trainable LoRA parameters and a modest amount of finetuning data can be enough. The source reports that for GPT-3, LoRA reached comparable or better performance than full finetuning on several tasks while training about 4.7 million parameters, roughly 0.0027 percent of the full finetuning count.

The same reasoning raises a question about pre-training itself. If factorization works so well during finetuning, could the model be factorized from the start? Low-rank pre-training could cut parameters and cost substantially, but full-rank pre-training still seems necessary to reduce intrinsic dimension to the point where low-rank methods work well. How much full-rank training is needed before switching is an open question.