Model Complexity and Training Data Variety

Suppose a model is performing poorly and you have a limited budget. A tempting idea is to take the rows you already have, copy them several times, and retrain. The dataset looks bigger, so surely the model gains something. Another common instinct is to gather a fresh batch of records that look almost identical to the old ones, on the theory that more rows always means more reliable predictions.

Both instincts rest on the same assumption: that the number of rows is what matters. This article corrects that assumption. The number of rows is only part of the story. What matters just as much is how varied those rows are, because variety is what determines how complicated a model you can safely use.

The central idea is a link between two things: how varied the training inputs are, and how complicated a model you can safely use. A model with many adjustable parts can bend itself to fit almost any set of points. That flexibility is useful when the training data covers many different situations, but it becomes a liability when the data is narrow.

If the training data is narrow, a flexible model can memorize the quirks of the few examples it has rather than learning a pattern that holds for new cases. This failure is called overfitting. The relationship can be stated directly: the larger the variety of data points in your dataset, the more complex a model you can use without overfitting.

Read the diagram from the data side. Dataset size usually brings more variety, and variety sets the ceiling on how complex a model can be. When a model's complexity goes past what the variety supports, it begins to overfit. This is why the number of rows alone does not settle the question: two datasets of the same size can support very different models.

A useful way to see the effect of variety is to follow a rule built from a small set of customer records. The rule, developed from a table of 12 customer rows, says that a customer older than 45 who has fewer than 3 children, or who is not divorced, is likely to want to buy a boat. The rule fits those 12 rows, but fitting 12 rows is weak evidence that it describes customers in general.

This also explains why more data usually helps. Collecting more data points will usually yield more variety, and larger datasets therefore allow building more complex models. The benefit of a bigger dataset is not that it is bigger. It is that a bigger, well-chosen collection tends to cover more of the situations the model will meet later.

The key distinction is between rows that repeat what the dataset already covers and rows that reach into regions of the input space that are not yet covered. Duplicates and near-duplicates add row count but no variety. A record that sits in an unexplored region adds both row count and new variety.

Read the diagram from top to bottom. Copying the dataset changes the row count from 100 to 1,000, but the covered input space stays the same. The model sees the same points more often, which does not tell it anything new about the pattern. A dataset that instead adds points in unexplored regions changes what the model has seen, and so it changes what complexity the model can safely support.

Near-duplicates behave the same way. A fresh batch of records that look almost identical to the old ones does not widen the range of inputs the model has learned from, so it does not justify a more complex model on its own.

When a model is not reliable enough, there are two broad levers. One is to adjust the model's settings. The other is to change the data by collecting more of it. The source makes a practical claim: having more data, paired with an appropriately more complex model to match it, can often produce large gains for supervised learning tasks.

The two levers change different things. Collecting varied records changes what the data can support, and that in turn determines how much model complexity is justified. Tuning changes how a given model fits the data it already has. Because dataset size is a condition you can influence in real projects, the source's advice is to take it seriously as a lever instead of assuming the model is the only thing to improve.

The ideas in this article are about supervised learning, where a model learns from examples that come with known answers. The rest of this chapter reviews the most popular supervised algorithms. For each one, the chapter describes how the algorithm learns from data and makes predictions, how model complexity plays out for that algorithm, the overall way it builds a model, and its strengths, weaknesses, and suitable kinds of data.

Many of these algorithms come in two variants, a classification variant and a regression variant, and both will be described. You do not need to memorize every algorithm on a first reading. Understanding how the models behave gives you a better feel for the different ways machine learning algorithms can work, and the chapter can later serve as a reference when you are unsure how a particular algorithm behaves.