Train and Test Splitting for Generalization

This article starts with a short program rather than a definition. Read it line by line and try to picture what each line does. The program trains a k-nearest neighbors classifier on labeled data, asks it for predicted labels on test points, and computes an accuracy value. Before reading on, write down what you expect each printed line to show and why.

The program prints two things: an array of predicted labels for the test points, and one accuracy value. Notice that the code assumes X_train, y_train, X_test, and y_test already exist, which means the data has been split into a training portion and a test portion. The rest of the article explains the mechanism that produces these two printed results.

The k-nearest neighbors algorithm is a classification method. To predict the label of a new data point, it looks at the training points that lie closest to that new point and uses their labels. It does not first build a set of learned rules. Instead, it stores the training examples and does its work when a prediction is requested.

The simplest version uses one neighbor. Imagine a new point sitting among labeled training points. You find the training point nearest to it, read that point's label, and assign the same label to the new point. With one neighbor, the model's behavior is fully determined by which training point is closest.

Finding the nearest neighbor is a search over the stored training points. For each new point, the algorithm measures how far it is from every training point and identifies the closest one. Distance is the measure of how similar two points are in their feature values: a small distance means the points are near each other in feature space. The ordering is decided by distance alone, before any labels are considered. Only after the nearest point is identified does its label matter.

Relying on only the single closest point can be fragile, so the algorithm can consider an arbitrary number k of neighbors instead. This is where the name k-nearest neighbors comes from. When more than one neighbor is considered, the algorithm uses voting: for each test point, it counts how many of its k neighbors belong to each class, and assigns the class that appears most often.

The same rule works with any number of classes. The algorithm counts the neighbors belonging to each class and predicts the most common one. Notice what the vote does not depend on: the value of k is the number of neighbors that vote, and it is not tied to how many distinct labels exist in the data.

The value of k is set when the classifier object is created, and it controls how many neighbors vote. Because the vote depends on which neighbors are included, changing k can change the predicted label for the same new point. The diagram below uses a hypothetical set of neighbors, ordered from nearest to farthest, to show this effect.

Trace the three rows. With k equal to 1, only the nearest neighbor (red) votes, so the prediction is red. With k equal to 3, the two additional neighbors are blue, so blue has two votes against one for red, and the prediction changes to blue. With k equal to 5, two more neighbors are added: one blue neighbor brings blue to three nearby votes... more precisely, the five neighbors contain three red and two blue, so red wins again. The same new point received three different predictions because the set of voters changed.

In scikit-learn, the algorithm is wrapped in a class called KNeighborsClassifier. The number of neighbors is set when the object is created. The fit call does not compute a boundary or a set of rules. For this classifier, fit stores the training dataset so that neighbors can be found later.

A useful way to read the four lines of the program is by when the work happens. Creating the object and calling fit only record information. The other two calls do the real computation. predict searches for neighbors and votes for each test point. score calls predict internally and then compares those predictions with the true labels to compute accuracy.

Each call has a distinct input. The constructor receives the setting n_neighbors. fit receives the training features and the training labels. predict receives only the test features and returns an array of predicted labels. score receives the test features and the true test labels, and returns a single accuracy value.

When an accuracy value looks wrong, the useful question is where the result diverged from what you expected. An accuracy number alone does not say which test points were wrong. The helper below compares each predicted label with its true label and reports the indices of the mismatches. It is a generated debugging helper written for this article.

Each index in the printed list points to one test point whose predicted label differs from its true label. Checking those specific points, rather than rerunning the whole program, shows what the model got wrong and makes the accuracy value interpretable. If you change n_neighbors, run this helper again and compare the two lists to see which points changed status.