Train and Test Splitting

KNeighborsClassifier predicts a class for a new point by looking at the training points closest to it. One setting controls how many of those neighbors take part in the vote: n_neighbors. A natural question follows immediately: which value should you choose, and what does that choice do to the model? Rather than starting from a rule, this article studies the question by running a loop that trains the classifier many times and records how well it performs each time.

The experiment uses the Breast Cancer dataset. The model is trained once per candidate value, and two numbers are kept for each value: accuracy on the training set and accuracy on a held-out test set. Reading those recorded numbers, and the plot they produce, is the skill this article builds.

The program first loads the dataset and splits it into a training set and a test set. The stratify argument keeps the proportion of each class similar in both sets, and random_state fixes the split so every run produces the same division of the data. Then the program tries every value of n_neighbors from 1 to 10. The listing below follows the steps described in the source.

Each pass through the loop does four things in order. It creates a new KNeighborsClassifier using the current value of n_neighbors. It fits that classifier on the training data only, which is where the classifier stores the training examples it will later compare new points against. It then computes the training score, and finally the test score. Because the model never saw the test data during fitting, the second score estimates how well the model handles new examples.

One pass of the loop builds a fresh model, trains it, and records two scores before the next value of n_neighbors is tried.

The two lists are filled in the same order the loop runs. The first entry of each list belongs to n_neighbors equal to 1, the second to 2, and so on. This alignment is what lets the plotting lines pair each score with the correct x-axis value. If you change the range, or add a step in the middle of the loop, the lists stay aligned only if every append happens inside the same pass.

The sweep records two scores for each value of n_neighbors, and they answer two different questions. The training score asks how well the model fits the examples it was built from. The test score asks how well the same model handles examples it never saw. Keeping these two questions separate is what makes the plot informative. A single score cannot reveal whether a model is memorizing its training data or generalizing beyond it.

When a dataset has exactly two features, the model's behavior can be seen directly. Imagine predicting a class for every possible point in the xy-plane, then coloring each region by the class that would be assigned there. The line where the color changes from one class to the other is the decision boundary. It is the divide between where the algorithm assigns class 0 and where it assigns class 1. The boundary is a property of the trained model, not of the data alone, so changing n_neighbors changes the boundary even when the training points stay exactly the same.

With a single neighbor, each test point simply takes the class of its one nearest training point, so the boundary follows the training data closely and wraps around individual points. As more neighbors are included, the vote is spread over more points and the boundary becomes smoother. A smoother boundary corresponds to a simpler model. Put in terms of complexity, few neighbors means high model complexity and many neighbors means low model complexity.

Follow this idea to its extreme. If n_neighbors equals the number of all points in the training set, every test point has exactly the same neighbors, namely all the training points. The vote is then identical for every test point, and every prediction is the class that is most frequent in the training set. The boundary disappears entirely because the model predicts one class everywhere.

The plot produced by the sweep puts accuracy on the y-axis and n_neighbors on the x-axis, with one line for training accuracy and one for test accuracy. Two failure patterns can be recognized in this single picture. Overfitting occurs when the model is too complex: it fits the training data very well but does worse on new data. Underfitting occurs when the model is too simple: it performs poorly even on the data it was trained on.

When you read a plot like this, follow the same order every time: start at the left edge, note the gap between the two lines, move to the right, and watch where the lines separate or both fall. The landmarks of the sweep are the points where you can see a pattern, not single numbers to memorize.

When the output of a sweep does not match what you expect, the sweep's data flow gives you a checklist. The data moves through four stages on each pass: split, fit, score, and record. A wrong picture usually comes from one of these stages, so check them in that order before blaming the model.