Understanding overfitting in classification models

Training a classification model produces a set of learned behavior. That behavior, however, does not tell you by itself whether the model is any good. A model can look reasonable on the very examples it was built from and still fail on examples it has never seen. This gap is the heart of overfitting, and it is the reason model evaluation exists.

Model evaluation is the set of steps that turns a trained model into a measured claim. It answers two questions: how often is the model right, and on what data was that measurement taken? The second question matters as much as the first. A measured accuracy computed on the examples the model learned from is not the same kind of evidence as one computed on examples it never saw.

This article follows one classifier, the k-nearest neighbors classifier, through the full path: how it predicts, how it is built and measured in Python, and what its accuracy number can and cannot tell you about the future.

The k-nearest neighbors classification algorithm has a local core idea. To predict the class of a new data point, it looks at the training points that lie closest to that new point and lets those neighbors inform the answer. Nothing global is computed, no equation is fitted to the whole dataset, and no rule is written down in advance.

The number of neighbors considered is controlled by a parameter, usually written as $k$. With $k=1$, the prediction is driven entirely by the single closest training point. With larger values of $k$, several nearby points contribute, and the predicted class reflects the neighbors together.

The diagram shows a one-dimensional case for clarity. The new point 3.0 has three nearest training points: 2.0 at distance 1, 1.0 at distance 2, and 8.0 at distance 5. The classes of those three neighbors decide the outcome. The source describes the general principle; the specific numbers here are an illustrative choice made for this article.

In Python, the algorithm is provided by a class named KNeighborsClassifier. A single class of this kind holds two responsibilities: the procedure that builds the model from training data, and the procedure that uses the built model to make a prediction. You work with the class in three stages. You first instantiate the class and set its parameters, then build the model by calling its fit method, and finally evaluate it by calling its score method.

Taken together, these steps form a short pipeline. The model is configured, built from training data, measured on test data, and then, if the measurement is satisfactory, put to use. The order matters. The test data must stay separate from the training data throughout, so that the measurement is an honest check rather than a rereading of material the model already studied.

Building the model means creating the object and then passing it two things: the training inputs, conventionally named X_train, and the training outputs, conventionally named y_train. The inputs describe each training example, and the outputs give the known class for each of those examples.

Notice that fit does not return a prediction. Its job is to build the model from the training pair. Evaluation and application are separate steps that come afterward. The nearest-neighbor model has no compact formula to memorize in advance; the stored training data is what the algorithm consults. That is why the training data must be supplied when the model is built, and why the same stored examples are needed whenever a prediction is made.

The code below follows the pattern described above. The data is split into a training portion and a held-out test portion, the classifier is created and fitted on the training portion, and then score is called on the test portion. The value 3 for the number of neighbors is an illustrative choice made for this example, not a value taken from the source. The variables X and y stand for your input measurements and their known class labels.

The score method computes the accuracy of the model. Accuracy is the proportion of predictions that match the true labels. Written as a ratio, it is the count of correct predictions divided by the total number of test examples.

To measure accuracy honestly, you give score a set of data the model did not learn from, along with the correct labels for that data. This is the test set. A model is always tempted to look good on the examples it was built from, so a separate test set is what reveals how the model behaves on examples it has not seen. Accuracy is only a trustworthy measure when it is computed on data the model did not learn from.

Once the measured accuracy is acceptable, the model can be applied to new data. In the source example, the new data consists of fresh flower measurements, meaning measurements that were not part of either the training set or the test set. The evaluation result is what gives confidence to take this step. The source states the expectation plainly: the model is expected to be correct about 97% of the time on such new data.

That expectation rests on assumptions. The test accuracy estimates performance only if the new measurements look like the data the model was evaluated on. The accuracy number is a measured claim about the test data, and it promises nothing beyond the conditions under which it was measured. It does not guarantee correctness on every future measurement, and it says nothing about data that comes from a different source or a different situation.