Data Preprocessing with StandardScaler

The data is split first, so the test set is set aside before any searching happens.

Programming

Why a Search Reports More Than One Number

A grid search tries many combinations of settings and reports which combination worked best. In this chapter the search covers a whole pipeline, so the candidates include a preprocessing step, a classifier, and the classifier's own parameters. Because of this, the output of a search is not a single number. It is a set of values that together describe which combination won, how well that combination scored during the search, and how it performed on data the search never saw. Reading those values correctly is a skill of its own, and this article builds that skill step by step.

What do you think happens?

The search produces one score while it runs and a second score afterward on the held-out test set. Should these two numbers be identical? If not, which one would you expect to be higher, and why?

Reveal answer

Answer:

They are not expected to be identical, because they are computed on different data. The score during the search comes from training data only, and the test-set score comes from data the search never used.

The cross-validation score is computed during fit on the training data, split into parts. The test-set score is computed once, afterward, on data that was set aside before any searching began. Since they answer different questions, there is no rule that one must be higher; the reason they differ is the subject of the rest of this article.

The lesson's code runs a grid search over a Support Vector Classifier (SVC) that is preceded by a StandardScaler, and then prints the three results that matter most: the best parameters, the best cross-validation score, and the score on the test set. The goal is not to run the code, but to understand what each printed line means before you trust it.

Tracing the Order of Operations

Notice the order of operations in the code. The data is split first, so the test set is set aside before any searching happens. The call to fit uses only the training portion. The test set appears only once, in the final score call. Keep this order in mind, because the rest of the article depends on it.

Which rows of the data produce the cross-validation score during the search, which rows produce the final test score, and at which step the two numbers part ways.

Read the diagram from left to right. The full dataset splits into a training portion and a test portion. The search works only on the training portion, dividing it into five parts because the code passes cv=5. The cross-validation score in best_score_ is produced during this stage. The test portion waits untouched until the final call to score, which is the only point where it enters the computation. The gap between the two numbers therefore arises in the structure of the sequence itself: they are produced at different stages, from different rows.

What best_params_ Contains

The attribute best_params_ is a dictionary. Each key names a step in the pipeline, or a parameter of a step, and each value is the setting that won the search. In this search the dictionary has four entries. Two of them hold whole objects, and the other two hold the specific numeric values that were tuned. The pieces are easier to read once you know what each key refers to.

Which object or step produces best_params_, which produces best_score_, and which method evaluates the best model on new data, and how the three outputs relate to each other.

A fitted GridSearchCV object reports three things. The attribute best_params_ names the winning combination. The attribute best_score_ holds the cross-validation score of that winning combination. The score method, called as grid.score(X_test, y_test), evaluates the search's best model on the held-out test data. The first two are stored results; the third is a computation you trigger, and it is the only one that touches the test rows.

python

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

pipe = Pipeline([
    ('preprocessing', StandardScaler()),
    ('classifier', SVC())
])

param_grid = {
    'preprocessing': [StandardScaler()],
    'classifier__C': [0.1, 1, 10],
    'classifier__gamma': [0.01, 0.1, 1]
}

grid = GridSearchCV(pipe, param_grid, cv=5)
grid.fit(X_train, y_train)

print(grid.best_params_)
print(grid.best_score_)
print(grid.score(X_test, y_test))

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

pipe = Pipeline([
    ('preprocessing', StandardScaler()),
    ('classifier', SVC())
])

param_grid = {
    'preprocessing': [StandardScaler()],
    'classifier__C': [0.1, 1, 10],
    'classifier__gamma': [0.01, 0.1, 1]
}

grid = GridSearchCV(pipe, param_grid, cv=5)
grid.fit(X_train, y_train)

print(grid.best_params_)
print(grid.best_score_)
print(grid.score(X_test, y_test))

Notes on the code

The data is split first, so the test set is set aside before any searching happens.
The pipeline names its steps; the classifier step is called 'classifier'.
Keys with a double underscore address a parameter inside a named step.
fit uses only X_train and y_train.
best_score_ is the cross-validation score; grid.score uses the held-out X_test and y_test.

This example is an original illustration written to match the structure of the lesson's code. The specific candidate values are chosen for teaching, and the printed numbers depend on the run, so the lesson does not claim any particular output. What matters is which line produces which value and which rows each line touches.

The Pipeline Inside Each Fold

The search does not scale the whole training set once and then compare classifiers. Because the pipeline is the object being searched, each candidate runs as a whole. For each fold and each parameter combination, the training part of the fold passes through the StandardScaler and then into the SVC. The held-out part of the fold must then be scored without being used to fit the scaler, so it is only transformed using what the scaler learned from the training part. The diagram traces one fold of one combination.

For one fold and one parameter combination, how data moves from the training part through the StandardScaler and the SVC, and where the held-out fold enters without being used to fit the scaler.

The held-out fold enters at the transform step and the predict step, but never at the fit step of the scaler. This is the reason a pipeline is safe to use in a search: the preprocessing is learned from the part of the data that the classifier is also learning from, and the held-out part is treated as unseen. The same structure is repeated for every fold and every combination, which is why the search is more work than a single fit.

Reading the C and gamma Grid

The param_grid dictionary lists the candidate values for each tuned parameter, and the search tries every combination of them. A grid of two parameters can be read like a table: each row is one value of one parameter and each column is one value of the other. Each cell is one combination, and the search records the cell that scores best. The grid below is an illustrative layout with generated candidate values, not the values from the source.

How each combination of C and gamma maps to one cell of the search grid, and which cell is recorded as the best parameters in this illustration.

Only the highlighted cell ends up in best_params_, and its values are the ones stored under classifier__C and classifier__gamma. The other cells were still evaluated during the search, but they do not appear in the printed result. When you read the output, you are reading the single winning cell, not the whole table.

Locating Where the Two Scores Diverge

The cross-validation score and the test-set score answer different questions. The cross-validation score asks how well the winning combination did on parts of the training data during the search. The test-set score asks how well the search's best model does on data the search never used. Treating these two numbers as the same kind of result is the most common reading error. Because they come from different rows at different stages, a gap between them is not automatically a bug.

When a result does not match what you expected, the useful question is where in the sequence the expectation and the result part ways. The code has a clear sequence of stages: split, search, report the search score, and report the test score. A mismatch can come from any stage, and each stage leaves a different trace in the output. The right approach is to check these stages in order: the data split, the fit call, the winning parameters, and then the comparison of the two scores.

Narrowing Down a Mismatch

Worked example

A Scenario With Two Scores That Disagree

A learner runs the grid search and sees a best_score_ that is noticeably higher than the output of grid.score on the test set. The learner wants to know where the mismatch comes from. This scenario is generated for teaching purposes.

1
Check the data split
Confirm that the test set was created before any call to fit. If the test rows had been included in the training data, the search would have seen them, and the test score would no longer be a fair held-out measure. In the order of operations shown in the lesson, the split happens first.
2
Check the fit call
Confirm that fit received only the training portion. The test set should appear only in the final score call. A fit call that used the full dataset would explain a gap for a reason unrelated to normal generalization.
3
Check the winning parameters
Read best_params_, starting with the double-underscore keys such as classifier__C and classifier__gamma. Confirm which preprocessing and which SVC settings won. This tells you which combination the test score is evaluating.
4
Compare the two scores
Only now compare best_score_ with the output of grid.score. The search score was computed on training parts, and the test score was computed once on held-out rows. A gap between them is expected to some degree, because they answer different questions. The learner's job is to decide whether the gap is explained by the stages above or whether one of the earlier checks failed.

Result

The learner finds that the split and fit steps were correct and that the winning parameters are the ones the search reported. The remaining gap is therefore a difference between two different measurements, not a sign that the test rows leaked into the search. The mismatch was located by checking the stages in order rather than by guessing.

Mistakes Learners Make When Reading Results

Common mistake
Treating best_score_ as the final test performance
What it looks like
Reporting best_score_ as the score the model achieves on new data.
Why it fails
best_score_ is the cross-validation score of the winning combination, computed during the search on training data only.
Do this instead
Report grid.score(X_test, y_test) for the held-out result and describe best_score_ as the search score.
Common mistake
Reading best_params_ as an explanation of the data
What it looks like
Concluding that a particular C value is the correct value for this dataset because it won.
Why it fails
The parameter values are evidence about which combination won the search. The search reports which settings worked best together, not why they work.
Do this instead
Describe the winning values as the best result of this search, and do not infer a general cause from them.
Common mistake
Reading the double-underscore keys as separate settings
What it looks like
Thinking classifier__C is a different parameter from the C inside the printed SVC object.
Why it fails
The double underscore separates the step name from the parameter name, so classifier__C refers to the same tuned value shown inside the SVC.
Do this instead
Look up the step name before the double underscore and the parameter name after it.
Common mistake
Fitting the scaler on the full dataset before the search
What it looks like
Calling StandardScaler().fit on all rows and then running the search.
Why it fails
The held-out rows then influence the preprocessing, so the search and test results are no longer cleanly separated.
Do this instead
Keep the scaler inside the pipeline so it is fitted only on the training part of each fold.
Common mistake
Assuming a mismatch between the two scores always signals a bug

Practice Reading a Printed Result

MEDIUM

Suppose a fitted search prints a best_params_ dictionary with entries for 'preprocessing', 'classifier', 'classifier__C', and 'classifier__gamma', and then prints a best_score_ value and a test-set score from grid.score. Explain which printed value was computed on training parts during the search, which was computed once on held-out rows, and which entries of best_params_ are the tuned numeric values. Then state where you would look first if the two scores differed more than you expected.

Hints

Ask which method call uses X_test and y_test.
Look for keys that contain a double underscore.
Work through the stages in order: split, fit, winning parameters, comparison.

Your attempt (not saved)

Summary of the Reading Strategy

Read a fitted search in the order its results were produced. Start with the split, confirm that the test rows were set aside, and then look at the winning combination in best_params_. Use the double-underscore keys to find the tuned values, and recall that C is the regularization parameter of the SVC and gamma is its kernel coefficient. Finally, compare best_score_ with the output of score, remembering that one comes from training data during the search and the other from held-out data.

At a glance

A fitted GridSearchCV object reports the winning settings in best_params_, the cross-validation score of the winning combination in best_score_, and a test-set score through the score method.
In a pipeline, best_params_ names the winning preprocessing and classifier steps, and double-underscore keys such as classifier__C identify the tuned values.
The cross-validation score is computed during the search on training data, while the test-set score is computed once on held-out data, so the two answer different questions.
When results do not match expectations, check the data split, the fit call, the winning parameters, and then the comparison of the two scores, in that order.

Related concepts

More concepts starting with D