Evaluating Model Generalization

Suppose you have built a classifier, tried many different parameter settings, and found one that scores 97% accuracy on the data you were checking against. It is tempting to write that number in a report as the model's quality. This topic explains why that conclusion can be wrong even when every individual calculation was done correctly. The problem is not arithmetic. The problem is which data was used to make the choice.

Parameter tuning is a loop. You change a setting, evaluate the model, compare the result with the others, and keep the best. Each pass through the loop is a separate attempt. When the evaluation step uses the test data, the test data becomes part of the training process, even though the model never fits its parameters to it directly. Every time a setting is kept or discarded based on test accuracy, a little information about the test data flows into the final choice.

Trace the arrows in the diagram. The test data reaches the comparison step on every pass, so the setting that survives is the one that fit the test data best. The final number that comes out of this loop has already been shaped by the test data. Nothing in that number tells you how the model behaves on data it has never been compared against.

The common belief goes like this: the model was tuned until it performed well on the test data, so its performance on the test data is a fair description of how it will behave later. Under that belief, the best test score is simply the model's true strength, and nothing more needs checking.

The belief breaks down because the score was not produced by a single fixed model. It was produced by a search. A score selected from many attempts is not the same kind of measurement as a score from one pre-decided model. When one attempt is picked because it looked best on the test data, the picking itself is part of the result, and the report usually hides it.

With many attempts, the highest test score is likely to be one where the chosen setting fit the particular quirks of the test set well. Those quirks are not guaranteed to appear in new data, so the score can overstate the model's ability to generalize. The source describes this as overfitting arising from parameter tuning: the model is shaped toward the test data rather than toward the underlying problem.

A judge is only useful if it has not already seen the evidence it is weighing. In a sound workflow, the training data fits the model, and a separate dataset that played no role in fitting or tuning gives the final verdict. In the flawed workflow, the test data is used for both the choosing and the judging, so the judge has already influenced the outcome it is meant to assess.

Once the test data has been used for tuning, it can no longer be used to assess the model. It has moved from being the judge to being part of the decision-making. Its later accuracy is no longer an independent measurement, because the settings were partly chosen to match it.

The key practical point is that tuning and judging must use different data. If the test set has already guided parameter selection, it can no longer serve as the unbiased assessment. An unbiased estimate requires a dataset that stayed out of every decision made about the model.

Two roles are easy to blur together. One dataset helps choose parameters, and another dataset gives the final score. The table below contrasts them using the roles described in the source: the data used to make choices must be kept separate from the data used to judge the result.

A clear way to see the problem is to follow the data step by step. The diagram below traces a sound workflow. Training data fits the model. Tuning data guides the choice of parameters. A separate evaluation dataset gives the final score. Read the arrows from left to right and notice that no dataset is used in two roles.

An independent dataset is one that was not used to create or tune the model. Once the parameters have been chosen, that dataset can measure performance without the bias introduced by the search. The earlier split into training and test sets was made for the same reason: a dataset that never influenced the model is needed to evaluate it.

Read the diagram as a checklist. The evaluation data must not feed the fitting relationship, must not feed the tuning relationship, and must not be a part of any decision about the parameters. When all of these hold, the evaluation data stayed out of every decision made about the model, and its score can be read as an unbiased estimate.