Model Evaluation with Cross-Validation

A very common first picture of machine learning is a single model that receives data and returns predictions. In that picture the model is the whole workflow. Once you have chosen a good model, it seems the remaining work is simply feeding it numbers. This belief feels reasonable, and it is exactly the misconception this article corrects.

Real-world machine learning rarely involves an isolated use of a model. Instead, it is a sequence of processing steps. Data is typically gathered, transformed, and shaped into useful inputs, and only then given to a model. Each stage depends on the one before it, so the workflow is a chain rather than a single action.

Three kinds of stages commonly appear in these chains: feature extraction, preprocessing, and models. Choosing the right combination of these stages is somewhat of an art and often requires trial and error. Because of this, the chain itself is the unit you need to manage, not just the model at its end.

The diagram below shows the general shape of a two-step chain: a scaling step (a preprocessing transformer) followed by a classifier (the model). Read it from left to right and notice that each stage receives the output of the stage before it. The tracing becomes concrete once you follow the training data through a fit call and then the test data through a predict call.

During fitting, the scaler sees the training inputs first, learns what it needs in order to transform them, and passes the transformed values forward. The classifier then fits on those transformed values, not on the raw inputs. During prediction, the new data goes through the same scaler and is then passed to the classifier. The scaler is not learned again at that point; it reuses what it learned during fitting.

The value of this design is that a whole chain of work becomes one thing. Your code no longer has to remember each stage separately. The pipeline holds the transformers and the final estimator together, and it exposes methods that you already use on individual steps.

The example below is an original illustration, not taken from the source. It chains a scaling step with a classifier. The scaler is the preprocessing step and the classifier is the model. It assumes X_train, y_train, and X_test already exist as your training inputs, training labels, and test inputs. The imports come from scikit-learn, the library that provides these classes.

Compare the pipeline with a hand-built workflow. Without a pipeline, you would fit the scaler on the training data, transform the training data, fit the classifier, and then separately transform the test data with the scaler before calling predict on the classifier. Every one of those separate lines is a chance to skip a step or run them out of order. The pipeline removes that bookkeeping, which reduces the likelihood of these mistakes and keeps the code more succinct.

Cross-validation evaluates a workflow by repeatedly fitting and testing it on different portions of the data. This is why the steps must live inside the pipeline. If a preprocessing step is fitted once on the full dataset before cross-validation begins, the statistics it learns include rows that will later serve as validation data. The validation data has then already influenced the transformation, and the evaluation becomes misleading. The source states that if some processing steps sit outside the pipeline, they may not be handled consistently across the repeated fits and tests, and the evaluation can then be misleading.

In the right-hand arrangement, the scaler is part of the pipeline. Each time cross-validation fits the workflow, the scaler is fitted only on that iteration's training folds, and the held-out fold is only transformed. The validation rows never shape the statistics used for their own evaluation.

In cross-validation, the data is divided into folds. In each iteration, one fold is held out as validation data and the remaining folds are used for training. The whole pipeline is then refit from scratch on the training portion, and its predictions are scored on the held-out fold. Repeating this for every fold gives one score per iteration, and those scores can be combined into an overall estimate.

Passing the pipeline itself to the evaluation function is what makes the scaling correct in every fold. The evaluation function can fit the whole chain on each training portion, so every transformer is learned only from data the model is allowed to see.

Grid search is described in the source as parameter selection: it tries combinations of settings and compares them. Each combination can be evaluated with cross-validation, which means each combination causes repeated fits and tests of the workflow. The source places special emphasis on this case because the repeated fits multiply any inconsistency. If a processing step sits outside the pipeline, it may be handled differently across these repeated fits, and the comparison between parameter settings becomes unreliable.