Model Merging

Finetuning is the step where a pre-trained model is adapted to a specific task or dataset. Many learners picture it as a full retraining run in which every number inside the model is nudged toward the new goal. That picture is accurate for the early days of finetuning, but for large models it is usually the wrong one. This article starts from that picture, shows why it breaks down, and then traces the efficient alternatives that lead into model merging.

Finetuning updates a pre-trained model's weights so that the model fits a specific task. In the early days of finetuning, the process looked much like pre-training, because both updated the model's entire set of weights. As models grew larger, updating everything became impractical for most practitioners.

The reasoning is a simple chain. Each parameter that gets updated during finetuning costs memory, so the more parameters a run must update, the more memory it needs. As models grew, that cost grew with them. The source notes that most practitioners do not have enough hardware, time, and data to finetune a foundation model in full.

Many later techniques share one motivation: reach strong performance while keeping the memory footprint small. Three named approaches attack this problem, and the most useful way to read them is by the lever each one pulls. One lever changes how many values are trained. The other changes how much space each value takes up. The diagram below places the three approaches against the shared bottleneck.

LoRA, which stands for Low-Rank Adaptation, is a PEFT technique. It adapts a pre-trained model by updating low-rank matrices rather than the model's full set of weights. The adjustment it learns is expressed compactly, using low-rank matrices, and it is best understood as a compact way of expressing an adjustment to a model rather than as a separate model built from scratch.

The source also connects LoRA to low-rank factorization, one of the fundamental ideas the chapter draws on. A detailed mathematical treatment is not given in the source material, so treat the low-rank idea as the reason LoRA can be compact and not as a procedure you can carry out from this article alone.

Once several finetuned models exist, a natural question follows: must each one be deployed separately, or can the useful parts be gathered into a single model? Model merging is the family of techniques built around that question. It combines multiple models into one model intended to work better than the individual models did on their own.

Notice what this definition does not require. It does not say the merged model must be trained from scratch on all the original data. It describes an outcome: one model that replaces several, with the aim that the single model does at least as well, and ideally better, across the work the separate models handled. The inputs to merging are models that someone has already finetuned.

The material lists several use cases for merging, including on-device deployment and model upscaling. On-device deployment connects directly to the memory concerns from earlier: a single merged model can be easier to store and run on a device than several separate models. The material names these use cases without detailing their procedures, so treat them as the motivations that make merging worth studying.

LoRA connects to this topic through modularity. Because each LoRA adaptation is a self-contained piece, multiple LoRA models are much easier to combine than full finetuned models. That modularity is one reason the idea of combining finetuned models became practical to explore.

TIES and DARE are two model merging techniques named in the material. The names themselves give a map of each method before you learn its details. TIES abbreviates TrIm, Elect Sign, and merge, so it lists three operations in order. DARE abbreviates Drop And REscale, so it names two actions followed by the combining of models.

Both methods work with task vectors, and both involve pruning them. A helpful way to hold them together is to treat them as two different answers to one design problem: how should the contributions of several finetuned models be selected and combined? TIES answers by trimming, electing signs, and merging. DARE answers by dropping and rescaling before the models are combined.

Pruning is the place where most readers' intuition about size goes wrong. The material makes an observation that reverses a common expectation: when task vectors are pruned, they become more sparse, but the finetuned model itself does not. Pruning in these merging methods is aimed at improving performance, not at reducing the size or inference latency of the finetuned model.