Sparse Models and Mixture of Experts

Suppose two language models come from the same family, and one is noticeably stronger on every task you try. What made the difference? A tempting answer is that the stronger one is simply bigger. That answer is partly right, but it hides several other forces that shape how well a model turns its design into useful behavior. This article walks through the main factors: how many parameters a model has, how those parameters are used, how much data it was trained on, what that data looked like, and how recently the model was built.

A model's parameters are the adjustable numbers that training shapes. When people compare models, the parameter count is often the first figure they look at, and it is common practice to write that count at the end of the model's name. Llama-13B, for example, identifies the Llama model family, developed by Meta, in its version with 13 billion parameters.

Parameter count sets a model's capacity to learn. Within one model family, a larger model generally performs better. That is the main reason size is a useful first comparison. But capacity is only the potential to learn. Whether that potential is realized depends on what the model is trained on and how training was done, which the rest of this article covers.

The parameter count also tells you roughly how much hardware is needed. Each parameter is stored as a number that takes a fixed number of bytes. If you know that number and the count, you can estimate the minimum memory required to hold the model while it runs inference, which means producing outputs rather than learning.

The byte size per parameter depends on the numeric precision used to store the values. Full precision (FP32) uses 4 bytes per parameter, half precision (FP16) uses 2 bytes, and 8-bit integers (INT8) use 1 byte. The table below applies the same 13 billion parameter count to each precision. These figures cover the stored weights only; the actual memory a running system needs can be higher.

One popular sparse design is the mixture-of-experts model, usually abbreviated MoE. An MoE model divides its parameters into separate groups, and each group is called an expert. The model does not run every expert on every token. Only a subset of the experts is active for processing any given token. Follow a single token through the layer in the diagram below to see which expert groups do the work and which stay idle for that token.

Read the diagram from left to right. The token arrives, and the routing step decides which experts should process it. In this illustration, two of the eight expert groups are active for the token, and the other six are left idle. Those idle expert weights still exist in the model, but they do no work for this particular token. A different token may be routed to a different subset, so the active experts can change from token to token.

A raw parameter count can be misleading when the model is sparse. A sparse model is one in which a large share of the parameters hold the value zero. Those zeros do not contribute to the model's computation in the same way that nonzero values do, so counting them as full parameters overstates the model's working size. The source states the consequence directly: a sparse model can require less compute than a dense one, even when the sparse model is larger on paper.

Mixture-of-experts designs work on a related principle. They decouple total parameters from per-token compute. The total parameter count describes what is stored. The per-token compute describes what runs for each token. The diagram below sets these two quantities side by side for a dense model and an MoE model.

The source's example is Mixtral 8x7B, which is built from eight experts, each with seven billion parameters. A naive calculation that multiplies those figures does not give the reported total, because some parameters are shared between experts, so the real count is lower. The important point is the division of labor. The parameters are split into experts, and only some of those experts are engaged for any token. The total and the per-token workload therefore differ.

Data quantity is the second major lever. Early on, researchers measured a dataset's size by counting training samples. That stops working for language models, because a training sample might be a sentence, a Wikipedia page, a chat conversation, or a whole book, and a book carries far more material than a sentence. The better yardstick is the number of tokens, since a token is the unit a language model actually operates on. Counting tokens tells you roughly how much the model has the opportunity to learn from its data.

Token counts depend on the tokenizer, because different models split text differently, and the same dataset can produce different token totals. Even so, token counts give a far more useful comparison than sample counts. Meta's Llama models show the trend the source describes, as the table below shows.

Modern language models are trained on datasets measured in trillions of tokens. The jump from one generation to the next shows how quickly the token budget has grown.

Two numbers are easy to confuse here, and the difference matters when you compare models. The first is the dataset's token count, which is how many tokens the collection contains. The second is the model's training token count, which is the total number of tokens the model processes during training. The source notes that training tokens count every pass over the data. A dataset is read once or several times, and each pass adds to the training total. The diagram below separates the two.

The two counts do not move together in a simple way. The source notes that training token counts do not scale neatly with parameter counts. A bigger parameter count does not automatically come with a bigger token budget. Chinchilla, with far fewer parameters than several of the other models in the source's comparison, was trained on a much larger number of tokens.

Capacity needs material to fill it. A large model that is trained on too little data can underperform a much smaller model that saw more. The source gives a deliberately extreme case. Imagine a 13-billion-parameter model whose entire training set is the single sentence 'I like pineapples.' Such a model has enormous capacity and almost nothing to learn from, so it will perform far worse than a much smaller model trained on a substantial body of text.

The lesson is that model size and data size have to be read together. Judging a model by its parameters while ignoring its training data gives an incomplete picture. Parameter count sets the capacity, and the training data decides how much of that capacity has anything to work with.

Quantity is not the only thing that matters. The source names three goals for training data, which it calls the three golden goals: quantity, quality, and diversity. Quantity has been the focus of this article so far, but a large dataset can still be a poor teacher if its contents are weak.

The source illustrates this with RedPajama-v2, an open dataset from Together that contains 30 trillion tokens. That is an enormous volume, but because it is made of indiscriminate content, the portion that counts as high quality is much smaller. Raw size and useful signal are different things. A dataset with many tokens and low quality may teach a model less than a smaller, cleaner, more varied one. The source points to a later chapter for a fuller treatment of quality and diversity.