Model Complexity and Algorithm Choice

Imagine you have studied a small table of customers and noticed a pattern that looks like a rule. Perhaps the rule says that older customers with few children, or customers who are not divorced, tend to buy a boat. Before you rely on that rule, you need to ask a basic question: how much evidence do you have that the pattern is real? The answer depends heavily on how many rows you examined.

The boat-buying example in the source uses a small table of 12 customer rows. From those rows, a rule was developed: if a customer is older than 45 and has fewer than 3 children, or is not divorced, then that customer wants to buy a boat. Now suppose 10,000 additional rows of customer data became available, and every one of them agreed with that rule. Think about how you would feel about the rule at that point.

A rule is a claim about the data. Each new row that matches the claim is another piece of evidence in its favor. When the rows are numerous and consistent with the rule, the claim becomes more believable. The reason is that the chance that the same agreement appeared by accident shrinks as the number of supporting rows grows.

The central idea is that confidence in a rule grows with the amount of data that supports it. Confidence does not jump to certainty in one step. Instead, it strengthens gradually as compliant rows accumulate. The diagram below traces this change for the boat example.

Notice what this idea does not say. It does not say that a large dataset proves a rule is true in every case. It says that the rule deserves more confidence than it would have with a small sample. Confidence and certainty are different, and keeping them apart is one of the main habits this chapter builds.

A pattern that looks strong in a few rows may be an accident of which rows happened to be sampled. The source's key point is that matching percentages are not enough to judge a pattern. The number of supporting rows matters just as much. A rule that matches 100 percent of a tiny sample can still be unreliable, while a rule that holds across a large number of rows earns more trust.

When a model does not perform as well as you hoped, there are two broad levers to pull. One is to adjust the model: change its settings, tweak its parameters, and tune it more carefully. The other is to gather more data. The source makes a practical point here. In many real situations you have some control over how much data you collect, and collecting more can be more beneficial than continued tweaking and tuning of the model.

The source also notes that this book works with datasets of fixed sizes. That is a teaching choice made to keep the examples manageable. It does not mean that dataset size is unimportant in practice. In the real world, the size of the dataset is often something you can decide to change.

The source connects two ideas: the quantity of data and the complexity of the model. It says that having more data and building appropriately more complex models can often work wonders for supervised learning tasks. The word appropriately matters. A model's complexity is not something to maximize. It should be chosen with the amount of data in mind, so that a larger dataset can support a richer model while a smaller one may not.

A flexible model has more room to match the particular rows it was trained on, including the accidental quirks of a small sample. The diagram below contrasts a simpler model with a flexible one when data is limited. This illustrates the general reasoning, not a measured result from the source.