Training a k-nearest neighbors classifier

Imagine you have a collection of past examples, each one labeled with its known class, and a new measurement arrives with no label. A natural way to guess its class is to look at the examples that are most similar to it and let them decide. That idea is the core of the k-nearest neighbors classification algorithm. Its prediction is local: it looks at the training points that lie closest to the new point and lets those neighbors inform the answer.

The algorithm has no fixed formula to memorize in advance. Instead, the stored training data is what it consults each time it makes a prediction. The steps run in order: the algorithm receives a new measurement, measures how far that point is from each training point, selects the k closest training points, and then takes a majority vote among their known classes. The class that wins the vote is returned as the prediction.

Because the stored training examples are needed every time a prediction is made, the training data must be supplied when the model is built. The model is not a compact summary that can be discarded after training. It is the examples themselves, together with the rule for consulting them.

The number of neighbors considered is controlled by a parameter usually written as k. When k equals 1, the prediction is driven entirely by the single closest training point. When k is larger, several nearby points contribute to the answer, and their classes are combined. The same new point can therefore receive a different predicted class depending on the value of k, which is why k is a setting you choose when you configure the model.

In Python, the algorithm is provided by a class named KNeighborsClassifier. A single class of this kind holds two responsibilities: the procedure that builds the model from training data, and the procedure that uses the built model to make a prediction. You work with the class in three stages. First you create the object and set its parameters. Then you build the model by calling its fit method with the training inputs and training outputs. Finally you evaluate it by calling its score method with test inputs and test labels.

Notice that fit does not return a prediction. Its job is only to build the model from the training pair. Evaluation and application are separate steps that come afterward. The same sequence can be read as a short pipeline in which the model is configured, built from training data, measured on test data, and then put to use if the measurement is satisfactory.

The score method computes the accuracy of the model. Accuracy is the proportion of predictions that match the true labels. To measure it honestly, you give score a set of data the model did not learn from, along with the correct labels for that data. This is the test set. Using a separate test set matters because a model is always tempted to look good on the examples it was built from. Training alone does not tell you whether the learned behavior is good; a model that looks reasonable on its own training examples may still fail on examples it has never seen.

Read this as a proportion. An accuracy of 0.9 means that nine out of every ten test predictions matched the true labels. The number answers the question of how often the model is right, but only on the data that was used to measure it. Model evaluation turns a trained model into a measured claim by asking two things: how often is it right, and on what data was that measured?

Once the measured accuracy is acceptable, the model can be applied to new data. In the source example, the new data consists of fresh flower measurements, meaning measurements that were part of neither the training set nor the test set. The evaluation result is what gives confidence to take this step. The source states the expectation plainly: the model is expected to be correct about 97% of the time on such new data.

Applying the model means passing new, unlabeled measurements to the model and using the predicted class it returns. Because those measurements were never used for training or testing, the test accuracy is the best available reason to trust the model, but that trust remains tied to the conditions under which it was measured.