Train-Test Splitting

Near the single-neighbor end, the training accuracy is perfect, but the test accuracy is below that of models using more neighbors, which marks the model as too complex.

Programming

An Experiment That Changes One Setting

This article begins with a short experiment rather than a definition. A KNeighborsClassifier is fitted on the Breast Cancer dataset many times. Each time, only one setting changes, the number of neighbors, and the model's accuracy is recorded on two different sets of data. Everything the article explains is a way of interpreting the numbers that this loop produces.

What do you think happens?

Across n_neighbors values of 1, 6, and 10, which setting gives the most complex model?

Reveal answer

Answer:

With one neighbor, each prediction follows its single nearest training point, so the model can capture the most fine local detail. A larger neighbor count averages over more points and produces a simpler model.

The guess you just made is the thread that runs through the rest of the article. The sections below explain what a decision boundary is, how the neighbor count shapes it, how to read the plot the loop produces, and how to trace an unexpected number back to the line of code that caused it.

Drawing the Decision Boundary

For a two-dimensional dataset, every point in the plane can be treated as a hypothetical test point. If you ask the classifier which class it would assign to each location, and color each location by that answer, the colored plane reveals a divide. This divide is the decision boundary: the line separating the region where the algorithm assigns class 0 from the region where it assigns class 1.

The boundary is not a fixed property of the data. It is produced by the classifier, and its shape depends on how many neighbors each prediction consults. The same training points can therefore yield very different boundaries depending on one setting.

How does the boundary between class 0 and class 1 change shape when n_neighbors goes from very small to very large?

Reading the three panels from left to right, the boundary moves from tightly wrapped around individual points to a smoother divide. With one neighbor, every location takes the label of its single nearest training point, so the boundary follows each example closely. With more neighbors, each location is labeled by a vote among several nearby points, so small irregularities get averaged out and the boundary becomes smoother.

Neighbors, Complexity, and Generalization

Definition

Model complexity describes how flexible a model is in fitting the patterns in its training data. For KNeighborsClassifier, the neighbor count controls this flexibility.

A small number of neighbors produces a model with high complexity, because its predictions can follow fine local detail. A large number of neighbors produces a model with low complexity, because its predictions depend on broad regions and change gently. So as n_neighbors decreases, the model's flexibility grows.

Flexibility matters because of what it does to generalization, meaning how well the model performs on data it has not seen during training. A very flexible model can fit its training examples closely without capturing a pattern that carries over to new data. A very rigid model may miss the pattern altogether. The neighbor count sits between these two failure modes, and the test data is what reveals which side of the balance a given setting falls on.

Reading the Accuracy Plot

The plot produced by the loop places n_neighbors on the horizontal axis and accuracy on the vertical axis. It has two lines: one for training accuracy and one for test accuracy. The gap between the two lines, and the position of each line along the horizontal axis, tells you where the model is too complex and where it is too simple.

At which n_neighbors values does training accuracy pull away from test accuracy, and which side of that gap is overfitting versus underfitting?

Read the diagram as a description of the plot. The single-neighbor setting sits at the complex end, where training accuracy is at its highest, yet test accuracy is lower than it is at more moderate settings. The middle region around six neighbors gives the best test performance. The large-neighbor end is too simple, and both accuracies drop.

Setting Up the Experiment Loop

The sketch below shows one way to structure the experiment. It is an illustrative example written for this article, not code taken from the source. It fixes the split once, then rebuilds the classifier for each neighbor count and records accuracy on both sets.

python

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)

for k in range(1, 11):
    clf = KNeighborsClassifier(n_neighbors=k)
    clf.fit(X_train, y_train)
    train_acc = clf.score(X_train, y_train)
    test_acc = clf.score(X_test, y_test)
    print(k, train_acc, test_acc)

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)

for k in range(1, 11):
    clf = KNeighborsClassifier(n_neighbors=k)
    clf.fit(X_train, y_train)
    train_acc = clf.score(X_train, y_train)
    test_acc = clf.score(X_test, y_test)
    print(k, train_acc, test_acc)

Notes on the code

The split is performed once, outside the loop, so every neighbor count is evaluated on the same test set.
A new KNeighborsClassifier is created on each iteration with the current value of k.
Training accuracy is computed on X_train and test accuracy on X_test.
range(1, 11) covers the neighbor counts 1 through 10 discussed in the plot.

Notice what the structure guarantees. Because the split is outside the loop, changing n_neighbors is the only thing that changes between iterations. Because a new classifier is created inside the loop, each recorded pair of numbers belongs to the setting printed beside it.

Mistakes That Corrupt the Reading

Common mistake
Changing random_state or stratify between runs and then comparing the accuracy numbers
What it looks like
Running the split once with random_state=0 and again with random_state=42, then treating the two accuracy curves as the same experiment.
Why it fails
Either option changes the test set, so differences in the curves may come from the split rather than from n_neighbors.
Do this instead
Fix random_state and stratify for the whole experiment, and change only the neighbor count.
Common mistake
Reusing one classifier across loop iterations
Why it fails
Each reading is supposed to reflect the neighbor count printed beside it. If the classifier is not rebuilt with the current n_neighbors, the numbers no longer match the setting being tested.
Do this instead
Create a new KNeighborsClassifier inside the loop, using the current value of n_neighbors.
Common mistake
Swapping X_train and X_test when scoring
Why it fails
Training accuracy would be reported as test accuracy and the reverse, so the gap between the two lines that defines overfitting would be misread.
Do this instead
Compute training accuracy on X_train and test accuracy on X_test.
Common mistake
Choosing the single-neighbor model because its training accuracy is perfect
Why it fails
Perfect training accuracy at one neighbor is the signature of overfitting. The source notes that its test accuracy is lower than the test accuracy of models using more neighbors.
Do this instead
Judge the setting by test performance, and look for the middle region where test accuracy is highest.

Tracing an Unexpected Result

Suppose you run the loop and the test accuracy curve peaks at a different place than the explanation predicts. Debugging means finding which line of the experiment produced the unexpected value. Work through the experiment in order, as the pipeline below shows, and check each step against what it is supposed to do.

Which step in the sequence of split, fit, and score could produce an unexpected accuracy number, and what would have changed at that step?

Check the first step if the test set seems to change from run to run, since changing random_state or stratify alters which points land in the test set. Check the second step if the curve seems to reflect a single model rather than a sweep, since a classifier that is not rebuilt for each setting does not reflect the current n_neighbors. Check the last step if the two accuracy lines seem swapped, since scoring the wrong split reports the wrong number.

MEDIUM

A colleague reports that test accuracy is highest at one neighbor and that the training accuracy line is flat across all settings. Using the pipeline, name the two steps you would check first, and explain what each check would reveal.

Hints

Ask whether the split was fixed for every setting, and whether the classifier was rebuilt inside the loop.
A flat training line may indicate the scoring call is reading the wrong data, so check which array is passed to score for training accuracy.

Your attempt (not saved)

Key Takeaways

A decision boundary is the divide between where the classifier assigns class 0 and class 1, made visible by coloring the plane with predictions.
Fewer neighbors gives a more complex model whose boundary follows training points closely; more neighbors gives a smoother, simpler boundary.
Overfitting appears when a complex model does well on training data but poorly on new data; underfitting appears when a simple model does poorly on both.
On the Breast Cancer experiment, performance peaks around six neighbors, and both extremes perform worse.
When a result looks wrong, trace it through the code step by step: check the split, the fresh classifier per setting, and which data is passed to each score call.

Related concepts

More concepts starting with T