Building Pipelines for Model Evaluation

You build a model, run cross-validation, and get a score that looks excellent. The natural reaction is relief, and the natural next step is to trust the number and move on. This article is about a case where that trust is misplaced. A score can be high not because the model has learned anything real, but because information from the held-out data found its way into the training process. When that happens, the estimate of how well the model will perform on new data is inflated.

The source builds its argument on a deliberately constructed example: a synthetic regression task with 100 samples and 1,000 features. Every feature is drawn independently from a Gaussian distribution, and the response is also drawn from a Gaussian distribution. Because the features and the target are generated independently, there is no real relationship between them. Keep that fact in mind throughout the article. Any honest evaluation of a model trained on this data should find nothing to learn.

The misconception this article targets is the assumption that feature selection is a harmless preprocessing step that can be done once, on the full dataset, before cross-validation begins. It feels reasonable. Feature selection is simply choosing a subset of relevant features, which seems like neutral housekeeping. Many people would not think twice about running it first and evaluating the model afterward.

The problem is that feature selection is not neutral with respect to the target. It looks at the relationship between each feature and the response in order to decide which features to keep. If that decision is made using every row, including the rows that cross-validation later treats as test data, then the test rows have already influenced which features the model sees. The evaluation is no longer a clean test of how the model handles unseen data.

The leak is a matter of ordering. In the flawed version, feature selection is fit on the full feature matrix and target before cross-validation splits the data into folds. The selected features are then passed to cross-validation, which trains a Ridge regressor on some folds and tests it on the remaining fold, repeating across the splits. Nothing in the code looks wrong. Each call does what it says. The flaw sits in the sequence.

Because the selection step has already seen the rows that will later serve as the test fold, the features it chose can be correlated with the target on both the training and test folds. Cross-validation then measures the model on data whose structure was partly shaped by the test rows. The result is an estimate that reflects the selection process rather than the model's ability to generalize.

This is why the effect appears even on random data. With 1,000 independent features and only 100 samples, some features will look related to the response by chance. Selection that uses all rows will keep those chance relationships, and because the test rows were part of that decision, the same chance relationships show up again on the held-out fold. The scores look like learning, but they are a record of the selection step.

The source's comparison with a pipeline gives a very different picture. Once selection is confined to the training folds, the estimate changes sharply. The reason is that features chosen using only training rows can look correlated with the target there, yet fail to be correlated on the test rows, because the data is random. The practical lesson is that the difference between a fully leaky evaluation and a clean one can be the difference between concluding that a model works very well and concluding that it does not work at all.

The first step in this example is a scaler and the last step is an SVM. When you call fit on the pipeline, it does not simply hand the data to the SVM. It passes the training data through each step in order, so that every step learns from exactly the data it will later process. Start with the code, then follow what happens to the data at each call. The distinction between which object receives training data and which receives test data is what the rest of the article depends on.

The source describes three actions inside a single call to fit. First, the pipeline calls fit on the scaler. Second, it transforms the training data with that scaler, producing scaled training data. Third, it fits the SVM using the scaled data. Each action depends on the one before it, so the order is part of the meaning of the pipeline.

The sequence matters because the SVM never sees raw training values. It sees the output of the scaler, which is exactly the representation that will be produced for test rows later. Keeping that chain together inside one object is what prevents the training and evaluation steps from drifting apart.

After fitting, you evaluate the pipeline by calling score on the test data. The source explains that calling score first transforms the test data using the scaler and then calls score on the SVM with that scaled test data. The pipeline does not refit the scaler on the test data. It applies the transformation it already learned during fit.

The source states that using a pipeline reduces the code needed for the preprocessing plus classification process. The bigger advantage is that the pipeline behaves as one estimator. Because of this, a single pipeline object can be passed directly to cross_val_score or GridSearchCV, rather than requiring you to coordinate a scaler and an SVM separately inside each evaluation loop.

In practice, the same pipeline object is reused across folds. For each fold, the selection or scaling step is fit on that fold's training rows, and the held-out rows are only transformed and scored. This is the mechanism that keeps the held-out fold out of the choice of features, which is exactly what the flawed workflow lacked.

Suppose a colleague reports a test score that is noticeably different from 0.95 for the same model. Rather than guessing, trace the pipeline stage by stage. The pipeline gives you a fixed order of events, so you can ask at each stage whether the data is what you expect. Work through the stages in this order, stopping at the first one that does not match what you expect.