Choosing the Number of Neighbors in k-Nearest Neighbors

Machine learning libraries make it tempting to call a few functions and accept whatever comes back. When a result is wrong, that habit makes the failure hard to diagnose. This article takes the opposite approach. You will read a short program, predict what it prints, and only then compare your prediction with the actual result. When the two disagree, the gap between your expectation and the output points you to the line where your mental model of the code broke down.

The model in this article is a k-nearest neighbors classifier. It is a classification algorithm that is easy to understand, and scikit-learn provides it. The whole idea is that a new data point receives the label of the training point it most resembles. Keep that idea in mind as you read the code.

In scikit-learn, every machine learning model is implemented as its own class, called an Estimator class. The k-nearest neighbors algorithm lives in the KNeighborsClassifier class, which belongs to the neighbors module. You cannot use the algorithm until you create an object from that class. Creating the object is also when you choose the model's parameters. The most important parameter is n_neighbors, the number of neighbors the model will consult.

The knn object you create is more than a name for the algorithm. It bundles together the procedure for building the model from training data, the procedure for making predictions, and any information extracted from the training data along the way. For KNeighborsClassifier, that extracted information is simply the training set itself. Calling fit is what hands the training data to this object.

Notice what fit changes and what it does not. It stores the training data inside the object. It does not produce a separate summary, a learned formula, or a set of weights. The object you created is the same object that fit returned, so the name knn and the name same_object refer to one model, not two separately trained models. The first printed line, True, confirms this.

A k-nearest neighbors classifier does not learn a formula or a set of weights from the training data. It answers each question by comparing the new point with the stored training points. The algorithm locates the training point closest to the new point and gives the new point that training point's label. This is the simplest version of the method, and it is the version the program above uses, because n_neighbors is set to 1.

The stored training set is consulted every time a prediction is made. This is why a model with no learned weights can still produce a prediction: the training data itself is the model. Every prediction is a lookup against that stored set, followed by a copy of a label.

The letter k generalizes the rule. Instead of consulting only the single closest neighbor, the method may consult any fixed number k of neighbors, such as the three or five closest training points. The prediction is then the majority class among those neighbors. When k is 1, the majority is simply the label of the one neighbor, which is why the program's answer depends on one training point alone. In scikit-learn, this number is the n_neighbors parameter.

The comparison above shows that changing n_neighbors changes which points vote, even when the final label happens to stay the same. With three neighbors, the two points at distance about 2.24 and 1.41 are both labeled 1, and only one vote goes to a label 0 point. Point (0, 1) is also at distance 5, tied with (1, 0), and both carry label 0, so the choice between them does not change the vote here. In general, a different k can change the answer, so the value is a decision that has to be made deliberately.

Read the program in that order. The construction line creates the object and fixes n_neighbors=1. The fit line stores the training data. The predict line compares X_new against the stored points and returns a label. If the printed value were not what you predicted, the first thing to check is which of these three stages produced a different state than you assumed. A useful habit is to print the parameters of the fitted object: printing a fitted model shows a long listing of its parameters, most of them defaults you did not set, and the n_neighbors=1 you did set should appear in that same list.