Evaluation Metrics for Supervised Models

Once a supervised model has been fitted, it is tempting to ask it how well it did on the same data it learned from, and to report that number as the result. The source material for this topic makes a precise point about why that number is not enough: a model evaluated only on the data it was trained on tells you about fit, not about generalization to new data. Fit describes how closely the model matches the rows it has already seen. Generalization describes how well it predicts rows it has never seen. Those are different questions, and a supervised model is usually built to answer the second one.

The standard remedy is to hold part of the data back before any fitting happens. A train-test evaluation has three distinct moves, and each has a specific job. First, the dataset is divided into a training set and a test set by train_test_split. Second, the fit method builds a model using only the training set. Third, the score method measures how the fitted model performs on the test set. The number you see at the end depends on all three moves, but only the third one touches the test data. Read the pipeline below from top to bottom and notice which steps touch the training rows and which touch the test rows.

The reason for splitting is that the training score is not the quantity we care about. The useful comparison is between the score on training rows and the score on test rows. A model that has memorized its training data will look excellent on those rows and noticeably weaker on the held-back rows, which produces a large gap between the two numbers. A model that generalizes shows training and test scores that are much closer together. The gap is therefore a diagnostic signal: it tells you whether the model's apparent quality survives contact with data it did not see.

To check the gap in the pipeline above, you can score the same fitted model on its own training rows as well as on the test rows. Both numbers come from the same model; only the rows differ. If the training number is high and the test number is much lower, the model fits its training data more closely than it generalizes. The source treats this as the central reason the held-back test set exists.

A single train-test split gives one estimate of generalization performance, and that estimate depends on which rows happened to land in the test set. A different random split could give a different number. Cross-validation is introduced as a more robust way to assess generalization performance. Instead of relying on one split, the data is divided into several parts, and each part takes a turn as the held-out evaluation data while the others are used for training. Every row therefore gets to serve as test data exactly once across the rounds.

Reading the rounds across the diagram shows the rotation. Each round holds out a different fold, trains on the remaining folds, and produces its own score. Because each fold appears as the test set in exactly one round, no single lucky or unlucky split dominates the estimate. The source introduces cross-validation at this level of motivation, so the key idea to carry forward is the rotation itself: the held-out role moves from part to part until every part has been tested.

Every supervised model has parameter settings that are chosen before fitting, and the choice affects how well the model generalizes. Grid search is presented as an effective method for adjusting these parameters in supervised models to reach the best generalization performance. The idea connects directly to everything above. The parameter choice should be judged by how it performs on held-out data, not by how it performs on the data the model was fitted to. Evaluation tells you whether a setting works, and parameter selection uses that evaluation to pick a setting.

In practice, grid search tries a list of candidate values for each setting, forms every combination of those values, and evaluates each combination with cross-validation. The combination with the best cross-validated result is selected. The procedure is simple to state, but it only works as intended when every candidate is judged on rows that were not used to fit that candidate, which is exactly what cross-validation provides.

The score method provides default measures. For classification it reports the fraction of correctly classified samples, which is accuracy. For regression the default measure is R squared. Treat the default score as a starting point rather than the complete verdict on a model. The source explicitly notes that the chapter goes beyond these default measures to discuss other ways of evaluating classification and regression performance.

A single number can hide differences that matter for a particular problem. Two classifiers with the same accuracy may behave very differently, so the choice of evaluation measure should match what you need the model to do. Accuracy counts all correct predictions equally. It does not tell you which kind of mistake the model makes, or whether those mistakes are acceptable for the task at hand. That is why the default score should be read together with an understanding of the problem, rather than accepted as the whole story.