Using Pipelines in Grid Searches

Many people who are new to grid search follow a sensible-looking routine. They scale their features first, using a scaler such as MinMaxScaler, and then run a grid search over an SVM's parameters on the scaled data. The reasoning feels clean: scaling is just a preparation step, so it should not matter whether it happens before or during the search. This belief, that preprocessing done outside the cross-validation loop is safe, is a common misconception, and this article explains why it fails and how pipelines address it.

Information leakage occurs when information from the test fold is used during preprocessing. In the scale-first workflow, the scaler is fit once on the full dataset before any cross-validation split is made. Its learned values then flow into every split. Each training fold and each test fold has already been transformed using statistics that came from the complete data.

Read the top diagram from top to bottom. The scaler's fit step touches every row, and that is the point where the test fold stops being separate. Everything downstream inherits that contact, so the scores at the bottom are computed on data that was prepared with knowledge of the held-out rows. The search is therefore not measuring generalization cleanly.

A pipeline changes the order of operations. Instead of scaling once up front, the pipeline puts the scaler and the SVC together so that the grid search treats them as one estimator. For each split in the cross-validation, the MinMaxScaler is refit using only the training splits. No information from the test split enters the parameter search through the scaler. The learned scale is therefore a product of training data alone for that split.

Walk through the loop above for a single split. The scaler is fit only on the training rows of that split. Those same training rows are then transformed with the learned scale and used to train the SVC. The test fold of that split is only transformed with the scale that was learned from training data; it is never used to fit anything. The next split repeats the whole sequence with its own training rows, so the preprocessing is refit for every split.

Once the preprocessing lives inside a pipeline, the parameter grid has to name the step that owns each parameter. The source uses a pipeline whose second step is an SVC named svm. Both parameters being searched, C and gamma, belong to that step. The syntax is the step name, followed by two underscore characters, followed by the parameter name. The double underscore separates the two parts, so the key for C is svm__C and the key for gamma is svm__gamma.

The step name is what lets grid search route each candidate value to the correct part of the pipeline. A plain key such as C, without the step name, does not say which estimator inside the pipeline should receive that value. This is why the source describes the modified syntax as a slight but necessary change from ordinary grid search.