Data Preprocessing and Scaling

Under this belief, the preprocessing is settled, and only the model's settings remain to be chosen.

Programming

A Common First Instinct About Scaling

A very common way to start a machine learning project goes like this: scale the data once, then search for the best parameters of the classifier. The scaling feels like a formatting step that happens before the real work begins. Under this belief, the preprocessing is settled, and only the model's settings remain to be chosen.

This article examines why that instinct is risky. The model evaluation and selection tools you have already met, grid search and cross-validation, are among the most useful tools a practitioner has. Their limitation is that, as described so far, they apply to a single supervised model. Many real models only perform well when the input data has been transformed first, so the scaler is part of what gets learned, not a fixed preliminary result.

EASY

Before reading further, think about this situation. You scale a dataset once with MinMaxScaler, then run a grid search over the classifier's settings using that already-scaled data. Which parts of the process were decided before the search began, and which parts did the search actually get to consider? Write down your guess before you continue.

Hints

Ask which values the scaler computed, and when it computed them.
Ask whether the grid search could ever change how the data was scaled.

Your attempt (not saved)

Following One Sample Through a Scaled Classifier

Consider what happens to one raw sample when a scaled classifier is trained. The sample begins as a row of raw feature values. The scaler transforms that row into a new row of values, and the classifier learns from the transformed row rather than the raw one. The output of the first step becomes the input of the second, and that is the whole idea of a chain.

What steps does a raw sample pass through in order, and where does each transformation hand its output to the next step?

The diagram shows the same chain as the code, with the fit and transform steps labeled. The scaler is not a decorative step at the front. It produces the representation that the SVM actually learns from, which is why it belongs to the model in a meaningful sense. Reading the chain from left to right, each arrow marks a point where one transformation hands its output to the next.

Why Grid Search Alone Does Not Cover Preprocessing

Definition

An algorithm chain is a sequence of processing steps and models in which each output feeds the next input. A chain for classification might be a scaler followed by a classifier.

Grid search and cross-validation, as introduced so far, were described for a single supervised model. They search over the settings of one estimator and score each candidate. When the model is only one link in a longer chain, the search needs to know about the other links too. The source describes many applications where the representation of the data matters greatly: the representation can begin with scaling the data and combining features by hand, and can extend to learning features with unsupervised methods. Most machine learning applications therefore need a chain of several processing steps and models working together, not one algorithm alone.

The source's own example makes the point. A kernel SVM trained on the breast cancer dataset performs much better when MinMaxScaler is applied to its features before training. The scaled SVM is the smallest useful chain: a transformer followed by a classifier. Once you have such a chain, the question of how to choose settings becomes harder, because there are now settings for the scaler as well as for the classifier.

The next part of the source introduces the Pipeline class, which is what makes it practical to apply grid search and cross-validation to such chains. This article only establishes why that tool is needed. Before the tool appears, it helps to understand the problem it solves.

What MinMaxScaler Learns From the Data

The reason the scaler cannot be treated as a fixed formatting step lies in what it is. MinMaxScaler computes its minimum and maximum from the training data it is fitted on, and then uses those values to transform any other data. Its settings are learned from data, which makes the scaler part of the model-building process.

Tracing the Scaled SVM in Python

The source follows a fixed order when it trains the scaled SVM: load and split the data, compute the minimum and maximum on the training data, rescale the training data, train the SVM, rescale the test data using the same scaler, and then score. The following code follows that order. It is an illustrative example written to match the steps the source describes.

python

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

scaler = MinMaxScaler().fit(X_train)
X_train_scaled = scaler.transform(X_train)

svm = SVC()
svm.fit(X_train_scaled, y_train)

X_test_scaled = scaler.transform(X_test)
print(svm.score(X_test_scaled, y_test))

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.svm import SVC

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

scaler = MinMaxScaler().fit(X_train)
X_train_scaled = scaler.transform(X_train)

svm = SVC()
svm.fit(X_train_scaled, y_train)

X_test_scaled = scaler.transform(X_test)
print(svm.score(X_test_scaled, y_test))

Walk through it

1
Lines 1 to 4 import the dataset loader, the split function, the scaler, and the SVM classifier.
2
Line 6 loads the breast cancer features and labels as arrays.
3
Line 7 splits the data so that some rows are never seen during training.
4
Line 9 fits the scaler on the training rows only, computing their minimum and maximum.
5
Line 10 transforms the training rows using those learned values.
6
Lines 12 and 13 create the SVM and train it on the scaled training rows.
7
Line 15 transforms the test rows with the same scaler, without refitting it.
8
Line 16 scores the trained SVM on the scaled test rows.

Notes on the code

The split happens first, so the test rows are held back.
The scaler learns its minimum and maximum from the training rows only.
The same fitted scaler transforms the test rows; it is not refitted.
The printed value is the score the chain achieves on the held-out rows.

What do you think happens?

If line 15 were changed to call fit_transform on the test rows instead of transform, which result would you expect the chain to be built on?

The test rows would be scaled with their own minimum and maximum, not the training values
The result would be identical because the scaler is always refitted anyway

Reveal answer

Answer:

The test rows would be scaled with their own minimum and maximum, not the training values.

A scaler fitted on one set of rows learns its values from those rows. Calling fit_transform on the test rows refits the scaler on test data, so the test rows no longer use the statistics learned from the training data. The source specifies that the test data is transformed using the same scaler.

Where the Scaler Sees the Held-Out Data

The scaler is the part of the chain most likely to see data it should not see. If the scaler is fitted before the split, its minimum and maximum are computed from every row, including the rows that will later be used for testing or as validation folds. The diagram below traces that path, which is the data leakage path the source warns about in its note that the scaler must be fitted on training data only.

Which rows does the scaler see when its minimum and maximum are computed, and do those rows include the rows that should stay hidden?

Before

scaler = MinMaxScaler().fit(X)
X_scaled = scaler.transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, random_state=0)

The scaler's minimum and maximum come from all rows, including those later held out.

After

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
scaler = MinMaxScaler().fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

The scaler's minimum and maximum come only from the training rows; the test rows are transformed with those values.

Where a Parameter Search Goes Wrong

Suppose you want better settings for the SVC. The natural first attempt is the naive one from the start of this article: scale the data once with MinMaxScaler, then run the grid search over the SVC parameters on the scaled data. The source states that this kind of naive approach may not work well. The problem is that the grid search can only vary the classifier. The scaling was fixed before the search began, so the search never considers whether a different scaling would serve the classifier better.

In each grid-search candidate and cross-validation fold, what gets refit, and how does that differ between preprocessing done once outside the loop and preprocessing done inside it?

The remedy the source gives is to combine the Pipeline class with GridSearchCV, so that the search can explore parameters for all processing steps at once rather than for the classifier alone. The top row of the diagram shows the fixed one-time approach; the bottom row shows the structure the remedy aims for, in which the scaler is refit inside each fold. The Pipeline class itself is introduced in the next chapter of the source, so only the reason for it is established here.

Mistakes Learners Make With Scaling

Common mistake
Treating the scaler as a formatting step that is finished before modeling starts
What it looks like
Scaling the full dataset once, then running a grid search over the SVC settings.
Why it fails
The scaler learns its minimum and maximum from data, so it is part of the learned system. A search over classifier settings alone cannot consider different scaling choices.
Do this instead
Treat the scaler and classifier as one chain and search over the chain as a whole, which is the purpose of the Pipeline class the source introduces next.
Common mistake
Fitting the scaler on all rows before splitting
What it looks like
```
scaler = MinMaxScaler().fit(X) followed by a train-test split of the scaled data.
```
Why it fails
The held-out rows influence the minimum and maximum used to train the model, so the held-out data is no longer hidden.
Do this instead
Split first, then fit the scaler on the training rows only, and use transform on the test rows.
Common mistake
Refitting the scaler on the test data
What it looks like
```
X_test_scaled = scaler.fit_transform(X_test)
```
Why it fails
The test rows get their own minimum and maximum, so they are no longer transformed by the same values the model was trained with.
Do this instead
Call transform on the test rows with the scaler that was fitted on the training rows.
Common mistake
Assuming a single scaling result carries over unchanged to every candidate setting of the classifier

Locating the Line Where Results Diverge

Return to the scaled SVM chain. Imagine a version of the code in which the scaler is fitted on X before the train-test split, and the SVM is then scored on the scaled test rows. Trace each step in order. The divergence from expectations is the line where held-out information first reaches the scaler's statistics, because every later score is then computed from a model whose scaling already depended on those rows.

MEDIUM

Using the scaled SVM code from earlier in this article, identify the single line that would make the test score unreliable if it were moved above the train-test split. Then explain in one or two sentences what the scaler would have learned from that line that it should not have seen.