Building Pipelines with Preprocessing Steps

Earlier in a machine learning workflow, you often tune the settings of one model and compare the results by hand. You try a few values, note which one scores best, and move on. That approach answers a narrow question: which hyperparameter values work best for this one model?

This article covers a more complete question. Instead of asking only which hyperparameter values work best, you ask which combination of model, preprocessing step, and hyperparameter values works best. The source works through this with the cancer dataset. It compares a Support Vector Classifier (SVC) against a RandomForestClassifier, and it also chooses between scaling the data with StandardScaler or leaving it unchanged.

The key idea is that GridSearchCV does not care what kind of thing it is searching over. A number, a model object, and a preprocessing step are all treated the same way. The search tries each candidate in the grid, evaluates it, and reports which one scored best. Once you understand how the grid is written, the rest of the workflow follows from it.

The central object in this workflow is the parameter grid. The source defines a grid with two entries. The first entry covers the SVC branch, and the second covers the RandomForestClassifier branch. Within each entry, the key 'classifier' holds the model objects to try, and the key 'preprocessing' holds the preprocessing choices. Other keys hold hyperparameter values for that model.

Notice that the two entries are independent. The search does not force the RandomForestClassifier to be combined with the SVC's gamma and C values. Each entry defines only the keys that apply to its own model. This is why a single grid can hold several different model families without mixing their hyperparameters together.

Keys such as classifier__gamma and classifier__C use a double underscore to point at a setting inside a pipeline step. The part before the double underscore names the step. The part after it names the parameter of that step. So classifier__C means the C parameter of whatever object is stored under the 'classifier' step.

The preprocessing key works differently. It is not a parameter of a model at all. It names a step, and its possible objects are listed directly. In the grid above, those objects are StandardScaler() and None. So a key with no double underscore lists whole step objects, while a key with a double underscore reaches into a step and names one of its parameters.

Before you run a grid search, it helps to know how much work it will do. Count the candidates in each entry of the grid, then add the entries together. The number of candidates in one entry is the product of the lengths of its value lists. Each key contributes one factor, and a key with a single value contributes a factor of one.

In this formula, N is the total number of candidates, e runs over the entries of the grid, k runs over the keys inside one entry, and |V_{e,k}| is the number of values listed under key k in entry e. Multiplication happens within an entry, because every combination of values inside that family is a candidate. Addition happens between entries, because the families do not combine with each other.

Once the grid is defined, the source runs the search in the usual way. The data is first split into training and test sets with train_test_split, using random_state=0 so the split is repeatable. The grid is then built around a pipeline object, which the source refers to as pipe. GridSearchCV is asked to use five cross-validation folds. Calling fit runs the whole search on the training data only. The test set is held back until the end.

It helps to trace what happens to the data when a candidate is evaluated. For each candidate, the data first passes through the preprocessing step, which is either StandardScaler or no change at all. The output of that step is what the classifier step receives. The classifier is then fitted and scored. This sequence is repeated for each fold, so every candidate passes through the same pipeline structure five times in this setup.

The printed output reports three different things, and each one answers a different question. The best_params_ attribute reports the winning combination across both models and the preprocessing step, not just the best hyperparameter value. In the source's run, the selected classifier is an SVC, the selected preprocessing is a StandardScaler, and the selected values are the ones stored under classifier__C and classifier__gamma. Reading this dictionary tells you which branch of the grid won, not only which number was best.

The best_score_ attribute reports the best cross-validation score from the training data. This is the score the search used to pick the winner. The score method, called on the held-out test set, reports how the chosen model performs on data the search never saw.

The two scores should be read together. When the cross-validation score and the test-set score are close, as they are in the source's results, that suggests the search did not overfit the cross-validation folds. A large gap between them would be a reason to be cautious about the reported performance, because the chosen model would look better on the data used to select it than on new data.