Tuning Hyperparameters in a Pipeline

A grid search over a pipeline needs a param_grid whose keys name a step and a hyperparameter, joined by a double underscore.

Programming

Why a Search Needs a Path Into the Pipeline

A pipeline is useful on its own, but one of its biggest payoffs comes when you search over its settings. Suppose you want to fit a LogisticRegression classifier on the cancer dataset, with a StandardScaler applied to the data first. You want to tune the regularization strength, and you want the scaling to happen inside every candidate so the search is honest. Everything in this article is about how that goal is written down in code.

The key idea is that the grid never touches the scaler or the classifier directly. You build the pipeline first, then write the grid against it, and only then fit the search. Everything the grid changes is addressed through a string key. Getting that key right is the whole problem, so before reading on, try to work it out yourself.

What do you think happens?

The classifier in the pipeline is a LogisticRegression, built with make_pipeline. What exact string should the grid key for its C parameter be?

C
LogisticRegression__C
logisticregression__C
logistic_regression.C

Reveal answer

Answer:

logisticregression__C

make_pipeline names each step after the lowercased class name, so LogisticRegression becomes logisticregression. The grid key joins that step name, two underscores, and the hyperparameter name C.

How make_pipeline Names Each Step

When make_pipeline builds the pipeline, it gives each step a name for you. The LogisticRegression step is named logisticregression, which is the lowercased class name. The name is what the grid uses to find the step, so it matters more than it first appears.

Because the names are created automatically, you should not assume what they are. Read them from the pipeline when you are unsure. The lowercased class name rule is the first thing to check, but the pipeline object is the authoritative source for the names it actually contains.

Anatomy of a Grid Key

The grid key has a fixed shape. It starts with the step name, then two underscores, then the hyperparameter name. For this pipeline the step is logisticregression and the hyperparameter is C, the regularization parameter of LogisticRegression. Written as one string, the key is logisticregression__C.

How the key splits into the step it targets and the hyperparameter it sets. The first part must match a step name in the pipeline, and the last part must match a parameter of that step's estimator.

Read the key from left to right. The left side must match a step in the pipeline, and the right side must match a hyperparameter of that step. The double underscore is the separator that tells the search where the step name ends and the parameter name begins.

Writing the param_grid

The source uses a logarithmic grid for C, covering values from $10^{-2}$ to $10^{2}$. Spacing candidates by powers of ten covers a wide range with only five values. The grid is a dictionary whose key names the step and parameter, and whose value is the list of candidates the search will try.

python

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))

param_grid = {"logisticregression__C": np.logspace(-2, 2, 5)}

grid = GridSearchCV(pipe, param_grid, cv=5)
grid.fit(X_train, y_train)
print(grid.best_params_)

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))

param_grid = {"logisticregression__C": np.logspace(-2, 2, 5)}

grid = GridSearchCV(pipe, param_grid, cv=5)
grid.fit(X_train, y_train)
print(grid.best_params_)

Notes on the code

make_pipeline creates the step named logisticregression from the LogisticRegression object
The grid key joins the step name, a double underscore, and the hyperparameter C
np.logspace(-2, 2, 5) produces five candidates spaced by powers of ten
cv=5 runs five cross-validation folds on the training split

Worked example

Building the grid for the scaled classifier

Write a param_grid that tunes C for the LogisticRegression step of a pipeline built as make_pipeline(StandardScaler(), LogisticRegression()).

1
Name the classifier step
The class is LogisticRegression, so make_pipeline names the step logisticregression.
2
Add the hyperparameter
C is the parameter to tune, so the key is the step name, two underscores, and C.
3
Choose the candidates
A logarithmic list from $10^{-2}$ to $10^{2}$ gives five values to try.
4
Assemble the dictionary
The key logisticregression__C maps to the list of five candidate values.

Result

param_grid = {"logisticregression__C": [0.01, 0.1, 1, 10, 100]}

Tracing Data Through a Grid Search

A pipeline passes data from one step to the next in a fixed order. In this example the scaler comes first and the classifier comes second. The grid search does not change that order. It changes only the classifier's C value on each candidate run, and the scaler stays in front of it on every run.

The training split enters the scaler first and then reaches the classifier. The grid search supplies one candidate C value per run and collects the scores that come back.

Two details matter when you trace this path. First, the grid targets a single named step, so a candidate value only ever reaches the step its key names. Second, each candidate is evaluated on the data as it moves through the steps that come before the named one, so the scaling is applied inside every candidate rather than once before the search.

The list in the grid expands into separate candidates, one per value. With cv=5, each candidate is also cross-validated across five folds of the training split. The search therefore runs the pipeline once per candidate per fold, and the scores from those runs decide which C value looks best.

Where a Mismatched Key Fails

The key must name a step that exists in the pipeline. Suppose you misspell the step as logisticregresion__C. The grid now refers to a name that no step carries, so no candidate value can reach the classifier. The grid key is the first place to look when a search does not behave as expected.

The grid is handed to the search, the search tries to apply the misspelled key to the pipeline, and the pipeline has no step with that name. The failure happens at the point where the search tries to set the candidate value.

Common mistake
Guessing the step name instead of reading it
What it looks like
Writing sklearn_lr__C because the estimator is a classifier you think of as a logistic regression model.
Why it fails
The step name is generated by make_pipeline from the class name, so an invented name matches no step.
Do this instead
Use logisticregression__C, or print the pipeline's steps to read the names it actually contains.
Common mistake
Misspelling the step name in the key
What it looks like
Writing logisticregresion__C with a missing letter.
Why it fails
The grid addresses one named step, and a misspelled name refers to nothing in the pipeline.
Do this instead
Copy the generated name exactly and check the key against the pipeline before fitting.
Common mistake
Dropping one of the underscores
What it looks like
Writing logisticregression_C with a single underscore.
Why it fails
The key shape requires the step name, then two underscores, then the hyperparameter name.
Do this instead
Use a double underscore between the step name and C.

Checking the Key Against the Pipeline

EASY

A colleague builds make_pipeline(StandardScaler(), LogisticRegression()) and writes the grid key as {"logisticregression__C": [0.1, 1, 10]}. Then they change the key to {"logistic__C": [0.1, 1, 10]}. Explain which key will reach the classifier step, and state where the mismatched version would fail. Identify the step name the grid must use.

Hints

Start from the name make_pipeline generates for LogisticRegression.
A key that names no step cannot reach any estimator in the pipeline.
Think about which call in the search tries to apply the key to the pipeline.

Your attempt (not saved)

MEDIUM

Write a param_grid for a pipeline built as make_pipeline(StandardScaler(), LogisticRegression()) that tries C values of 0.01, 0.1, 1, 10, and 100. Then trace the path a single candidate value takes from the grid to the classifier, naming each object it passes through in order.