Dataset and Model Evaluation

Building a machine learning model is only half of the job. A model can produce answers, but the practical question is how well it will perform on data it has never seen. A model that has memorized its training examples may look impressive on those same examples and still fail when it meets something new. This article walks through the standard approach: hold part of the data back, build the model from the rest, and then measure performance on the held-back portion.

The dataset is divided into two parts that play different roles. The training set is used to build the model. The test set is kept aside and used only to evaluate how well the model generalizes to new, previously unseen data. Each example belongs to one part or the other, so the two roles stay separate.

The diagram shows the split as a one-way division: the dataset feeds two separate destinations. Notice that the test set does not appear on the path that builds the model. The model is constructed from training examples alone, and the test examples enter only at the evaluation step.

The separation is what allows the test set to act as an honest check. Nothing in the test set is used to construct the model. If test examples leaked into building the model, the resulting score would describe data the model had already partly learned from, and it would no longer be a fair measure of performance on unseen inputs.

The source uses the k-nearest neighbors classification algorithm as its example model. It makes a prediction for a new data point by looking at its closest neighbor or neighbors in the training set. The prediction therefore depends on which known examples sit nearest to the new point. Because the algorithm is a classification method, its output is a class label rather than a continuous number.

The source describes the workflow in four moves: create the classifier object and set its parameters, build the model by calling fit with the training data and training outputs, then call score with the test set data and test set labels. The source names the parameters X_train and y_train for the training inputs and outputs. The listing below is a generated illustration of that same workflow in Python. It is not the source's own listing, and the parameter value shown is a choice made for this illustration. Assume the data have already been divided into the training and test parts.

Accuracy is the fraction of test examples the model classifies correctly. The source treats a result of about 97 percent as meaning the model is correct 97 percent of the time on the test set. Written as a ratio, the definition is shown below.

The left column describes what the number measures: the proportion of correct predictions on held-out examples. The right column names what the number does not promise. A score on one test set describes that test set, and the source frames any expectation about future correctness as an estimate carried forward from it.

Once the test set result looks acceptable, the model can be applied to new data. In the source's example, the new data are new flower measurements. The source reports that the evaluation gave enough confidence to apply the model to these measurements and to expect it to be correct about 97 percent of the time. The reasoning is that the test set was a stand-in for unseen data, so its accuracy is the best available estimate of accuracy on future inputs.

This is why a model that performs well on a test set can be trusted, within limits, on new predictions. The test examples were never used to build the model, so their results are not inflated by memorization. The trust rests on the assumption that future inputs resemble the test inputs.