Mixture-of-Experts Models

Suppose you have two trained models, and each one is good at a different part of the same job. The natural question is whether you could combine them into a single model that is better than either one. This article covers that idea, which is called model merging. The hub this material belongs to is named Mixture-of-Experts Models, and the source material for it is organized around merging.

Merging is easiest to understand by comparing it with finetuning. Finetuning changes one model so that it becomes better at a particular job, and it does this by updating that model's weights with new data. Model merging takes a different route. It starts with several existing models and produces a new model by combining them. That new model is the custom model you end up using. The flexibility of this route is the main reason practitioners reach for it.

The purpose of merging is to produce one model that is worth more than the models it came from. The first kind of added value is better quality. The second is a smaller footprint: two models that each do a different task can sometimes be merged into one model that does both tasks with fewer parameters. This is especially attractive for adapter-based models, where two adapters trained on the same base model can be folded into a single adapter. Merging also supports multi-task finetuning. Instead of training one model to handle several tasks through more involved routes, you can build those capabilities by combining separately produced models.

Every merging technique answers one question: how should the parameters of the constituent models be combined? Summing is the answer that works directly on weight values. Each weight in the merged model is computed from the matching weights of the constituent models. Two summing methods are widely used, linear combination and spherical linear interpolation. Both assume the components being merged line up with each other, so the first practical step is to check their scales. If one model's parameters are far larger than the other's, you can rescale the models before summing so their values fall into a similar range.

Linear combination is a weighted average. For a single pair of matching weights, one from model A and one from model B, the merged weight is a blend controlled by a mixing value alpha. The expression below captures that blend.

Linear combination works best when the models share a base. Its meaning is clearest in that situation, because the two models then differ only in what their finetuning added. The next worked example uses small numbers to trace a single pair of weights through the blend. The numbers are illustrative and are not taken from a real model.

Spherical linear interpolation, or SLERP, is the second summing method. Instead of a straight-line blend, SLERP follows the shortest path between two vectors on a sphere. Its result is therefore not simply the midpoint of the two values, and the path it traces differs from a straight-line average. SLERP merges exactly two models at a time. If you need to combine more than two models, you must do it in steps rather than in one SLERP operation.

Summing is only one of three broad answers to the merging question. Layer stacking takes whole layers from different models and places them one after another. Concatenation joins parameters together so that the merged component holds all of them. These are not mutually exclusive, and a single merge may sum some layers while stacking others. The three approaches differ most in how much they change the size and shape of the result.

Summing keeps the architecture because every merged weight is a blend of corresponding weights. Layer stacking can change the architecture, since it rearranges whole layers. Concatenation is the opposite stance from summing. Rather than blending parameters, it joins them, so the parameter count of the merged component is the sum of the parameter counts of all its constituent components.

Layer stacking is also called passthrough or frankenmerging. It builds a merged model by taking different layers from one or more models and placing them on top of one another. For example, you might take the first layer from model 1 and the second layer from model 2. A well-known early example is Goliath-120B, which was built from two finetuned Llama 2-70B models, called Xwin and Euryale. The merge took 72 of the 80 layers from each model and combined them. The trade-off is that stacked models usually need further finetuning before they perform well, unlike models produced by summing.

Linear combination takes on its clearest meaning when the models share a base. Suppose you finetune a base model for one specific task. If you subtract the base model's weights from the finetuned model's weights, the difference is a vector that captures what that finetuning added. This difference is called a task vector, and it is also known as a delta parameter. When you finetune with LoRA, you can construct the task vector directly from the LoRA weights.

Once you have task vectors, you can do arithmetic on them, and this is called task arithmetic. Adding two task vectors combines the capabilities they represent. Subtracting a task vector reduces a capability. Subtraction has a practical use: it can remove behaviors you do not want, such as invasive capabilities like facial recognition or biases picked up during pre-training.

Finetuning adjusts many parameters, but most of those adjustments are small and do little for the task. An adjustment that contributes nothing to performance is called redundant. Yadav and colleagues showed that a large portion of a task vector can be reset to zero with minimal loss in performance. Resetting a parameter means returning it to its base-model value, which sets its task vector entry to zero.

Redundant values are harmless when a model is used alone, but they can cause trouble after merging. Merging techniques such as TIES and DARE therefore prune redundant parameters from task vectors before combining them. Both papers reported that this step can noticeably improve the quality of the merged model. The benefit grows with the number of models being merged, because each additional model gives redundant parameters another chance to interfere with the other tasks.