Data Quality in Model Training

Imagine you build a chatbot and test it carefully. It answers polished, well-formed questions beautifully. Then real users arrive. Some write long messages stuffed with background details. Others type two-word commands. Some misspell every third word, and some ask questions in a programming language the team never considered. The model may suddenly look much weaker than it did in testing. This article is about why that happens and what a team can do about it.

The central question is how much variety a training set needs in order for a model to handle the situations it will meet after deployment. The answer leads to a concept called data diversity, and to two neighboring ideas that are easy to confuse with it: data quality and data mix.

A model can only generalize from what it was trained on. Training data is therefore the model's entire experience of language. If that experience contains only one way of asking for things, the model has no evidence that other ways of asking refer to the same problem.

Real users rarely ask questions in one standard form. The same request can arrive as a careful paragraph with references or as a terse fragment, and it can contain typos. If the training data only contains the polished version, the model has seen one style of asking and may not recognize the others as the same kind of problem. Coverage therefore depends on sufficient diversity in the data, and that requirement is why the attribute is commonly named data diversity.

Diversity is not a single fixed target. It depends on the application. Not every application needs the same kind of variety, and two systems show why.

Diversity can be measured along several axes. One of them is domain diversity, which concerns how topics or domains are represented in the data. The source notes that its meaning shifts across the phases of building a model, so a single fixed mix is not automatically right at every stage.

The source's table of high-level domains across training phases shows that the best mix differs by phase. General English knowledge stays large in every phase but climbs sharply during preference finetuning, while math, reasoning, and coding shrink. The table lists only high-level domains, so finer topics such as geometry within math do not appear in it.

Data quality and data diversity are separate properties. Quality concerns accuracy and cleanliness. Diversity concerns variety. A dataset can score high on one and low on the other, and the two need not move together.

Dataset A is a set of carefully checked, accurate answers that all address the same narrow kind of question. It is high in quality but low in diversity. Dataset B is a large pile of varied but sloppy text, with many errors and irrelevant fragments. It is diverse but low in quality. Neither set alone is ideal. A model needs data that is varied enough to cover its users and accurate enough to teach correct behavior.

The clearest evidence for needing both properties comes from a controlled comparison. Researchers Zhou and colleagues, reported in 2023, finetuned a 7-billion-parameter language model on three datasets. Each dataset contained the same number of examples, 2,000, but they differed in character. The first was high-quality but not diverse. The second was diverse but low-quality. The third was both diverse and high-quality. Because the size was held fixed, any difference in the resulting models could be attributed to the character of the data.

The model finetuned on the dataset that was both high-quality and diverse outperformed the models finetuned on either of the other two. Holding the example count constant is what makes this result informative. Quality and diversity each contributed something different, and the combination mattered more than either property alone.

Once a team accepts that diversity and quality both matter, the next question is how to combine different kinds of data in practice. The combination is called the data mix. The right data mix depends on the specific application and use case, so no universal recipe exists.

The diagram is a generated reasoning aid, not a fixed rule from the source. The source describes two broad approaches to filling in the data mix, and the first is the simplest. Choose a mix that accurately reflects how the application is really used. If most real traffic is customer-service questions, the mix should look like customer-service questions.

The more principled approach uses experiments to find the best mix. Meta ran scaling law experiments for Llama 3. For each candidate mix, the team trained several small models on that mix and used their results to predict how a large model would perform on it. The final mix was the best estimate suggested by those experiments.

A common question is how many examples a team needs. The source answers by analogy: asking how much data you need is like asking how much money you need. The answer varies enormously. At one extreme, a fun experiment by Jeremy Howard and Jonathan Whitaker showed that a language model can learn something useful from a single example. At the other extreme, some teams have finetuned models with millions of examples.

Pre-training a foundation model from scratch uses far more text. The source gives Llama 2 as trained on 2 trillion tokens and Llama 3 on 16 trillion tokens. If each example were 2,000 tokens long, those totals would correspond to about 1 billion and 15 billion examples, respectively. A dataset of millions of examples is therefore small by comparison, which is one reason finetuning on top of a pre-trained model is popular.

It is tempting to assume that a team with millions of examples should simply start from a pre-trained model and finetune it. The source cautions that the team should evaluate whether training from scratch would improve performance. Finetuning on top of a pre-trained model is typically more efficient, but there are situations where finetuning can be worse, especially when a team has a large amount of training data.

In the typical case, the pre-trained weights adjust to the finetuning data. In the ossification case, the pre-trained weights resist adjustment and end up adapting poorly. The source adds that smaller models are more susceptible to ossification than larger ones, so model size is one factor a team should weigh when choosing between finetuning and training from scratch.