Model Complexity and Generalization

A common first instinct when building a model is to ask how well it performs on the examples it was built from. If the score on those examples is high, it is tempting to conclude that the model is good. This article examines why that conclusion can be wrong, and why the real goal of a machine learning model lies somewhere else.

Earlier in this module you saw that KNeighborsClassifier predicts a class for a new point by looking at the training points closest to it. One setting controls how many of those neighbors take part in the vote: n_neighbors. A natural question follows immediately. Which value should you choose, and what does the choice do to the model? Rather than starting from a rule, this article answers that question by running a loop that trains the classifier many times on the Breast Cancer dataset, and then by learning how to read the scores it records and the plot they produce.

Seen this way, training performance is only a starting point. A model that memorizes the examples it was shown has achieved something, but not the thing we actually need. The question that matters is whether the patterns it learned still hold when it meets cases it has never seen before.

Complexity is best understood as a dial rather than a yes-or-no property. A model with little capacity can only represent simple relationships in the training data. A model with greater capacity can follow more intricate variations. Complexity is not good or bad in itself. What matters is whether the amount of complexity suits the problem and the data at hand.

Complexity is something you adjust, not a fixed feature you simply accept. The source material notes that every algorithm covered in its discussion offers ways to control model complexity. Adjusting it wrongly is what leads to the two failure modes, underfitting and overfitting, which sit at opposite ends of the dial. Both describe a mismatch between the model and the training data, but in opposite directions.

The reason training performance is not enough becomes clear once you follow the data. The training set is what the model is fitted on. The test set is held back, and the model never sees it during fitting. A score on the training set therefore measures how well the model reproduces what it was shown. A score on the test set estimates how the model will handle examples it has not encountered, which is the situation that matters after deployment.

The program below loads the Breast Cancer dataset and splits it into a training set and a test set. It then tries every value of n_neighbors from 1 to 10. For each value it builds a new classifier, fits it on the training data, and records two numbers. Finally it plots both lists against n_neighbors. The stratify argument keeps the proportion of each class similar in the training and test sets, and random_state fixes the split so that every run produces the same division of the data.

Each pass of the loop produces two numbers, and they answer different questions. The training accuracy is computed on the same examples the model was fitted on, so it measures how well the model reproduces data it has already seen. The test accuracy is computed on held-out examples, so it estimates how the model behaves on new data. Both are fractions of correct predictions, and both are recorded in the same pass, using the same fitted classifier.

When a dataset has exactly two features, you can see the model's behavior directly. Imagine predicting a class for every possible test point in the xy-plane, then coloring each region of the plane by the class that would be assigned there. The line where the color changes from one class to the other is the decision boundary. It is the divide between where the algorithm assigns class 0 and where it assigns class 1.

The boundary is a property of the trained model, not of the data alone. Changing n_neighbors changes the boundary even when the training points stay exactly the same. The shape depends heavily on how many neighbors vote, and the two extremes show the range clearly.

With a single neighbor, the boundary follows the training data closely, wrapping around individual points, because each test point simply takes the class of its one nearest training point. As you consider more neighbors, the boundary becomes smoother. A smoother boundary corresponds to a simpler model. In these terms, few neighbors means high model complexity, and many neighbors means low model complexity.

Follow the idea to its extreme. If n_neighbors equals the number of all data points in the training set, every test point has exactly the same neighbors, namely all the training points. The vote is then the same for every test point, and every prediction is the class that is most frequent in the training set. The boundary disappears entirely because the model predicts one class everywhere.

The plot produced by the sweep puts accuracy on the y-axis and n_neighbors on the x-axis, with one line for training accuracy and one for test accuracy. Overfitting occurs when the model is too complex, so it fits the training data very well while doing worse on new data. Underfitting occurs when the model is too simple, so it performs poorly even on the data it was trained on. The sweep lets you recognize both in one picture.

Read the plot from left to right. The left edge, where n_neighbors is 1, shows the most complex model. The right edge, where n_neighbors is 10, shows the simplest model in this sweep. The best performance in this experiment lies between these extremes, around six neighbors.

When the output does not match expectations, check the loop first. The place where a sweep usually goes wrong is the step that records the scores, since a misplaced append breaks the alignment between each score and its n_neighbors value. The plotting code is only as correct as the lists it receives.

The flowchart below follows one full run of the loop. Trace the steps in order, then notice where the two scores are stored and where the loop decides whether to continue.

The discussion in this chapter ties complexity to the settings you choose when you apply an algorithm. For many algorithms, choosing the right parameters is important for good performance. Parameter settings are one of the main levers for controlling complexity, so they connect directly to underfitting and overfitting. An algorithm's assumptions must be understood before it is applied.

Representation matters as well. Some algorithms are sensitive to how the input data is represented, and in particular to how the features are scaled. An algorithm that behaves well on one representation may behave poorly on another, even when the underlying data is identical.