Hyperparameter Tuning for Logistic Regression

This article starts with code rather than theory. The program below uses a logistic regression model on the cancer dataset, wrapped in a Pipeline and tuned with GridSearchCV. Read it once now, then run it through the trace that accompanies this article and watch what each line does before reading the explanation that follows.

The three lines at the centre of the workflow are the split, the construction of the search object, and the call to fit. The search object is built from three ingredients: the pipe, which is the estimator to tune; the param_grid, which lists the hyperparameter values to try; and cv=5, which sets the number of cross-validation folds. The setup above is only illustrative. What matters is the shape of the workflow, not the particular values chosen for C.

GridSearchCV performs a grid search over the hyperparameters you specify for a given estimator, and it performs cross-validation while doing so. Because the estimator here is a Pipeline, each candidate setting is applied to a model that already includes the data processing step. The scaler and the classifier travel together through every candidate evaluation, so the pipeline is the unit being tuned.

Read the diagram from top to bottom. GridSearchCV hands the training data to the pipeline. The pipeline sends that data through the scaler first, then passes the scaled values to the logistic regression. The score that comes back is what the search uses to compare candidates. The sequence matters: the classifier never sees raw values when the pipeline is doing its job, because the scaler runs before it.

A grid is a list of candidate values for each hyperparameter, and the search tries them one at a time. In the illustrative grid above, the single hyperparameter C has four candidate values. Each candidate is evaluated with cross-validation, and cv=5 means each candidate is scored across five folds of the training data. The search then keeps the candidate with the best cross-validated result.

After the search has finished, grid.best_estimator_ holds the whole fitted pipeline with the best settings found. The source prints this object to show which configuration won. The printed result is a Pipeline, and its representation lists every step together with the arguments it was built with. Reading that printout is the fastest way to confirm what the search actually selected.

The hierarchy shows where to look. The pipeline is the outer object, and it contains named steps. The tuned value C is not stored on the pipeline itself; it belongs to the logistic regression step, which is why reading the printout and reaching into the individual step are both necessary when you want to check the tuned value.

Once the search has finished, the logistic regression inside the best pipeline can be reached by its step name. The named_steps attribute of a Pipeline is a dictionary-like view of the steps, keyed by the names you gave them. In the setup above, the classifier was named logreg, so it is retrieved with that key. The coef_ attribute of the logistic regression then holds the fitted coefficients.

Coefficients represent the change in log odds, so they are the quantities you interpret when explaining the model. Note that the coefficients belong to the classifier as it was fitted inside the tuned pipeline, so they reflect the scaled inputs the classifier actually received, not the raw measurements.

When a tuned result does not match what you expected, the useful question is where in the workflow the mismatch first appears. An error in an early stage often shows up only later as a strange coefficient or an unexpected winning value. So the method is to walk the stages in order and check the earliest stage that could explain the symptom.

Work through the stages in that order. Begin with the split, then check the construction of the pipe and the param_grid, then the call to fit on the training data, then the printed best_estimator_, and finally the coefficients pulled from named_steps. A symptom seen at the last stage, such as a strange coefficient, may have started at an earlier stage such as a wrong split or a wrong grid.