Model Evaluation using Accuracy

Imagine you have a collection of labeled examples, and you are handed a new data point whose label is unknown. How would you guess that label? One simple idea is to look at the examples that are most similar to the new point and let them decide. The k-nearest neighbors classification algorithm works exactly this way. It makes a prediction for a new data point by considering its closest neighbor or neighbors in the training set.

This article follows one complete path through that idea. First you will see how the algorithm uses neighbors to make a prediction. Then you will see how the KNeighborsClassifier class carries out that work through its fit and score methods. Finally you will see how the accuracy of the resulting model is measured on the test set, and what a value such as 97% actually tells you.

The algorithm does not need to find a rule that separates the classes in advance. There is no step where it first studies the whole training set and writes down a boundary between the labels. Instead, when a new point arrives, the algorithm looks at the training examples nearest to that point and uses their labels to decide the prediction. The work happens at the moment a prediction is needed, by consulting the neighbors.

In Python, the k-nearest neighbors algorithm is implemented in the KNeighborsClassifier class. This class contains two things: the algorithm that builds the model, and the algorithm that makes a prediction using that model. The class therefore plays two roles, one for building and one for predicting.

You do not write the neighbor search yourself. Instead you create an object from the KNeighborsClassifier class and call its methods. The object is the thing that holds the work: once it exists, you use its fit method to build the model and its score method to evaluate that model.

The workflow has a clear order. Training data goes into fit, which produces a model. The test set then goes into score, which reports accuracy. The same model is never asked to be scored on the data it was built from in this workflow; the test set is kept separate so that the reported accuracy describes how the model handles data it has not seen during fitting.

A percentage such as 97% is easiest to understand as a count of matches. Each test example gets a predicted label from the model, and that predicted label is compared with the true label of the same example. Where the two agree, the prediction counts as correct. Where they disagree, it counts as a mistake. The score is the share of test examples that were predicted correctly.

The source states that the score method computes accuracy on the test set. So when a result reads 97%, that number describes the test set predictions, not the training examples the model was fitted on. It tells you how often the model's predictions matched the true labels for test examples that were kept apart from the training data.