Parameter Selection using Grid Search

When you tune a model, the natural picture is a search over the model's own settings. You prepare the data once, perhaps transforming it or extracting features from it, and then you run a search that varies only the model. The processing steps feel like a fixed preface that has nothing to do with the search. The source for this topic explains why that instinct leads to trouble, and why the Pipeline class is the tool that corrects it.

Follow what happens to a single preprocessing step, here a scaler, when it is applied to the full data set before the data is split. The source states that a processing step left outside the repeated refitting loop means the evaluation no longer measures the workflow you actually plan to use. Trace the path below and notice where the test rows enter the picture.

In this path the scaler has already seen the test rows before the split happens. Its statistics were shaped by data that the evaluation is supposed to hold back. The score on the test rows therefore no longer represents what happens when a truly new data point arrives. The mistake lives in where the step sits, not in the model itself. The same kind of error occurs whenever a step is prepared once, outside the loop that repeatedly refits and scores the workflow.

The Pipeline class is described in the source as a general-purpose tool for chaining together multiple processing steps in a machine learning workflow. Its purpose is to package those steps so they behave as one unit. The source describes the Pipeline as a single Python object with the fit, predict, and transform interface. From the outside, you call the same methods you would call on one model, and the object passes the work along through its steps.

The diagram shows the chain as one fixed path. Because the steps are connected in a fixed order, the chain applies transformers in the right order every time. Every new data set passes through the same path, including the test set, so no step can be skipped by accident. The source notes an important consequence: an evaluation procedure does not need to know how many steps the chain contains. It calls the same methods on the pipeline, and the pipeline passes the work along. The source also notes that this allows writing more succinct code.

Cross-validation and grid search both repeat work. Cross-validation refits the workflow on different portions of the data, and grid search tries different parameter settings. The source says that in both cases the evaluation repeatedly refits the workflow on different portions of the data, and that cross-validation and grid search need the whole chain so every step is evaluated consistently. Trace the nesting below to see exactly what gets rebuilt and in what order.

The loop has a specific meaning when you compare several models or processing choices. Each pass rebuilds the whole chain with a different set of choices, so the comparison is always between complete workflows. A candidate is never judged by its model alone while its preprocessing is silently held fixed. This is the reason the source points to pipelines for parameter selection: the candidates differ as whole workflows, and each candidate is simply another version of the same chained object.

The source says that building the chain by hand makes errors easy to introduce. The errors it names are forgetting to apply all transformers on the test set, and not applying them in the right order. Both happen precisely when processing is treated as separate from the evaluation. The following mistakes show how each one arises and how a single pipeline object guards against it.