Class Imbalance in Training Data

When people judge a training dataset, they often start with a simple question: how much data is there? Size matters, but it is not the whole story. Data coverage is about whether the training data spans the kinds of inputs and behaviors a model will meet, not only how much data exists. A dataset can be large and still leave whole areas of the problem untouched.

This article follows a model through a small thought experiment. You will trace what the model has seen, find where its experience runs out, and then see how new data can be generated to extend that experience. Along the way you will also meet class imbalance, the situation where one label has many examples and another has very few.

Start with the team's 1,000 customer messages. When they sort the messages by length, the pattern is clear: most fall in the middle range, a few are very short, and none are very long. Each time the team asks what the model has learned, the honest answer is that it has learned about middle-length messages. The very short and very long regions are empty.

Notice what this walkthrough reveals. The coverage gap is defined by a characteristic of the inputs, here their length, rather than by the total count. A dataset of a thousand examples can still have a hole in it. Improving coverage therefore means finding the specific characteristics that are missing and adding data that carries them.

The question this article asks is practical: when the data you already have does not cover an area well enough, can you create new data that does? The source answers yes. Data synthesis generates data with targeted characteristics, either to improve performance or to get a model to express specific behaviors. The method is to choose the characteristic first and then generate data that carries it.

The source gives two concrete illustrations of this targeting. In the first, you generate very short texts or very long texts, which fills the length gap from the previous section. In the second, you create conversations that contain toxic phrases for a toxic detection model, so that the detector sees the kind of content it must recognize.

Targeting also works in the opposite direction. The source notes that if real-world data is toxic, you can synthesize safe data instead. The goal is then not more examples of a problem but examples of the desired behavior that the real data lacks. Whether you want more of a characteristic or less of it, the underlying move is the same: decide what the data should look like and generate toward that description.

Synthetic data can be used for two different purposes that are easy to confuse. In one, the data is built to show a model a behavior you want it to express. In the other, the data is built to challenge the model, and such data is called an adversarial example. The source defines an adversarial example as synthetic data designed to challenge a model's performance, and notes that AI is commonly used to produce them.

A concrete case from the source makes the adversarial idea clearer. Gekhman et al. (2022), in their work on TrueTeacher, used large language models to generate factually inconsistent summaries. Those summaries were then used to train models to detect factual inconsistency. The synthetic data is deliberately wrong in a specific way, and that deliberate wrongness is exactly what the detector needs to learn to notice.

The source also describes a broader use. Perez et al. (2022), in Discovering Language Model Behaviors with Model-Written Evaluations, discussed data synthesis techniques for generating datasets that test 154 different AI behaviors. These included personality traits, political views, ethical stances, and social biases. The authors reported that in head-to-head comparisons, language-model-generated datasets approached the quality of human-written ones, and sometimes exceeded them.

Coverage problems are not only about the shape of the inputs. They can also be about how labels are distributed. Class imbalance describes the situation where classes in the data are unequally distributed: one class has many examples and another has very few. A model trained on such data sees the common class constantly and the rare class seldom, so the rare class is the one it is most likely to be underprepared for.

The source states that it is possible to generate data for the rare class to address the challenges of class imbalance. The intended effect is that the rare class gains examples so that the classes are represented more evenly. The source does not prescribe a particular target ratio, so the right amount of generated data depends on the task.