Finetuning Versus Training from Scratch

Imagine you build a chatbot and test it carefully. It answers polished, well-formed questions beautifully. Then real users arrive. Some write long messages stuffed with background details. Others type two-word commands. Some misspell every third word, and some ask questions in a programming language the team never considered. The model may suddenly look much weaker than it did in testing. The problem is often not the model architecture. It is the data the model learned from, which did not reflect the messy variety of the people who would actually use it.

This article looks at that gap. The central question is how much variety a training set needs in order for a model to handle the situations it will meet after deployment. The answer leads to a concept called data diversity, and to two neighboring ideas that are easy to confuse with it: data quality and data mix.

Before defining anything, trace what the model actually saw. In the chatbot story, the training data contained only the polished version of each request. The model therefore learned one style of asking. When a long, rambling, or misspelled request arrives, the model may not recognize it as the same kind of problem it already handles well. The gap is between the situations represented in training and the situations the model meets after deployment.

Notice what the diagram leaves out. The model's architecture is not the variable here. The same architecture trained on a different set of examples would behave differently on the messy requests. This is why the source locates the problem in the data.

The reasoning behind the definition is simple. Real users rarely ask questions in one standard form. The same request can arrive as a careful paragraph with references or as a terse fragment, and it can contain typos. If the training data only contains the polished version, the model has seen one style of asking and may not recognize the others as the same kind of problem. Coverage therefore depends on sufficient diversity in the data, and that requirement is why the attribute is commonly named data diversity.

Diversity can be measured along several axes. One of them, domain diversity, concerns how topics or domains are represented in the data. Other axes also exist beyond domains. These include the number of tokens in the context and the response, the number of conversational turns, and the proportion of human-written data compared with AI-generated data.

Not every application needs the same kind of variety. The right amount and kind of diversity is determined by what the system is supposed to do. The source contrasts two systems to make this concrete.

Diversity is not the same as quality. Quality concerns properties such as accuracy and cleanliness of the data. Diversity concerns the variety of what the data covers. A dataset can score high on one and low on the other. The two properties are separate, and a team should check both.

Consider a thought experiment. Take a set of carefully checked, accurate answers that all address the same narrow kind of question. The data is high in quality but low in diversity. Now take a large pile of varied but sloppy text, with many errors and irrelevant fragments. That data is diverse but low in quality. Neither set alone is ideal. A model needs data that is both varied enough to cover its users and accurate enough to teach correct behavior.

Once a team accepts that diversity and quality both matter, the next question is how to combine different kinds of data in practice. The combination is called the data mix. The right data mix depends on the specific application and use case, so no universal recipe exists.

The source describes two broad approaches. The simplest is to choose a mix that accurately reflects how the application is really used. If most real traffic is customer-service questions, the mix should look like customer-service questions. A more principled approach uses experiments to find the best mix.

In one example, Meta ran scaling law experiments for Llama 3. For each candidate mix, the team trained several small models on that mix and used their results to predict how a large model would perform on it. The final mix was the best estimate suggested by those experiments.

A common question is how many examples a team needs. The source answers by analogy: asking how much data you need is like asking how much money you need. The answer varies enormously. At one extreme, a fun experiment by Jeremy Howard and Jonathan Whitaker showed that a language model can learn something useful from a single example. At the other extreme, some teams have finetuned models with millions of examples.

Put the millions of examples in perspective. Pre-training a foundation model from scratch uses far more text. The source gives Llama 2 as trained on 2 trillion tokens. If each example were 2,000 tokens long, that total would correspond to about 1 billion examples. The source gives Llama 3 a much larger token count, which points the same way. A dataset of millions of examples is therefore small by comparison, which is one reason finetuning on top of a pre-trained model is popular.

It is tempting to assume that a team with millions of examples should simply start from a pre-trained model and finetune it. The source cautions that the team should evaluate whether training from scratch would improve performance. Finetuning on top of a pre-trained model is typically more efficient, but there are situations where finetuning can be worse, especially when a team has a large amount of training data.

The two paths start from different places. Finetuning begins with a pre-trained model, and the question is how its existing weights adjust to the application data. Training from scratch begins with an untrained model, and all of what it learns comes from the data it consumes. In the typical finetuning case, the pre-trained weights adjust to the finetuning data. In the ossification case, the pre-trained weights resist adjustment and end up adapting poorly.