Weight Parameters and Model Structure

Imagine you already have a large model that has been trained before. It knows a lot in general, but you now need it to perform well on one specific task, and you only have a modest amount of data for that task. Adapting the model to that data is called fine-tuning. The question this article answers is how that adaptation can be done without paying the full cost that fine-tuning usually demands.

The technique this course introduces for that purpose is LoRA. Its name stands for Low-Rank Adaptation. Before looking at either word in the name, it helps to see the problem it targets: the computational costs and resources that fine-tuning a large model normally requires. Large models are expensive to adjust in full, so the goal is a method that changes the model in a much more limited way while still producing a task-specific result.

The key phrase in that definition is only a small subset. The model's weights are not all treated the same way under LoRA. Instead of changing the entire set of weight parameters to match the new data, LoRA restricts the changes to a small portion of them. The pretrained model remains the starting point, and the adaptation is focused on that limited part. Think of the pretrained weights as a fixed foundation and the LoRA parameters as a small, trainable addition built on top of it.

The diagram shows the division of labor. The pretrained weight matrix W is not the thing being trained. The matrices A and B are the only ones that receive updates during training, and the adapted model is the result of combining the frozen base with what those two matrices have learned. The diagram describes the standard LoRA arrangement, in which the low-rank update is kept separate from the original weights.

Large language models contain many layers, and each layer is built from weight matrices. Suppose a large weight matrix $W$ is associated with a specific layer. LoRA can be applied to all linear layers in an LLM, but it is easier to understand the idea by focusing on a single layer. Everything that follows about one layer applies in the same way to each linear layer where LoRA is used.

Keep one matrix in view as you read the rest of the article. Call it $W$. In a standard fine-tune, $W$ itself would be changed during training. Under LoRA, $W$ stays as it was in the pretrained model, and the learning happens in a separate, much smaller set of parameters attached to that same layer.

Read the diagram from left to right. The input enters the layer along two routes. One route passes through the frozen matrix W, which produces the base output that the pretrained model would have produced. The other route passes through the trainable matrices A and B, which produce the learned adjustment. The two results are added together to give the layer's output. Only the second route is affected by training, so every update lands in the small trainable part while the pretrained base remains intact.

The word low-rank refers to a mathematical idea: limiting model adjustments to a smaller dimensional subspace of the total weight parameter space. You can think of the full weight parameter space as the complete set of directions in which the weights could possibly move during training. A smaller subspace is a restricted set of those directions. Adjustments made within that restricted set are the only ones LoRA allows.

Why would restricting the directions be a good idea? The source explains that this restriction effectively captures the most influential directions of the weight parameter changes during training. In other words, the changes that matter most for the task are kept, while the space of possible changes is made smaller. This is the reason the approach can be both focused and efficient.

In the standard LoRA formulation, the full weight matrix $W$ has shape $d \times d$ in the case discussed here. Rather than learning an update of that full size, LoRA learns two thin matrices. Matrix A has shape $r \times d$ and matrix B has shape $d \times r$, where $r$ is much smaller than $d$. Multiplying B by A produces an update of the same $d \times d$ shape as $W$, but every such update must pass through an $r$-dimensional bottleneck. That bottleneck is the smaller subspace the word low-rank describes.

The benefit of the low-rank structure becomes concrete when you count the trainable parameters. The source describes the savings in general terms, as a significant cut in computational costs and resources. The count below uses a made-up but representative size to show where those savings come from.

Notice what the example does and does not show. It shows that the trainable portion shrinks dramatically when r is much smaller than d. It does not change the pretrained weight matrix W, which still has its full size and still participates in every forward pass. The savings are in what is trained, not in what the model contains.

The main reason LoRA is useful and popular is that it enables efficient fine-tuning of large models on task-specific data. Efficiency here means that the approach significantly cuts down on the computational costs and resources that fine-tuning usually requires. Large models are expensive to adjust in full, so a method that touches only a small part of the parameters is valuable when the goal is a specific task rather than a complete rebuild.

This fits the situation described at the start of the article: a large pretrained model, a specific task, and only a modest amount of data for that task. Training a small set of adapter parameters means fewer values have to be computed and stored during learning. The source does not spell out the memory or overfitting consequences, but it is reasonable to read the restriction as a way of focusing learning on the task without rewriting the whole model.