Feature Scaling and Model Performance

Imagine a dataset where each row describes one example and each column is a feature measured on its own scale. One feature might sit between 10 and 15, while another sits roughly between 1 and 9. Nothing is wrong with either feature on its own. The problem appears when the values of different features live in different ranges, because the features are then not on equal footing. Many machine learning methods treat a larger numeric spread as a stronger signal, so the scale of a feature can influence the result for reasons unrelated to what the feature actually means.

Preprocessing is the set of transformations applied to data before it reaches a learning method. This article looks at four transformations that appear together in scikit-learn: StandardScaler, RobustScaler, MinMaxScaler, and Normalizer. All four change the numbers in a dataset, but they do so for different goals. The useful question is not which one is best in general, but which property you need the transformed data to have.

Keep one small two-feature dataset in mind as the running picture. The first feature stays between 10 and 15, and the second between 1 and 9. The question to hold onto is this: when you transform the data, what exactly is guaranteed afterward, and what is left unguaranteed? Each scaler gives a different answer, and reading the four answers side by side is the fastest way to build an intuition for them.

Read the table by column to see each feature's own range, and by row to see one complete data point. Both views matter, because the four scalers differ in which direction they operate. Three of them work on each feature (column) separately, while the Normalizer works on each data point (row) as a whole.

StandardScaler transforms each feature so that its mean is 0 and its variance is 1. It does this one feature at a time, so every column ends up with the same center and the same spread. The practical result is that all features come to the same magnitude. No single feature dominates merely because its raw numbers were larger.

In the running dataset, Feature A and Feature B both end up with mean 0 and variance 1 after StandardScaler. Their minimum and maximum values, however, are not forced to 0 and 1. If you check the transformed table, you will find values on both sides of zero, and the largest value for a feature may be well above 1. This is the expected behavior, not an error.

RobustScaler aims for the same goal as StandardScaler, which is putting features on a comparable scale, but it builds that scale from different statistics. Instead of the mean and variance, it uses the median and the quartiles. The median is the middle value of a feature, and the quartiles mark the spread of the central portion of the data. Because these statistics describe the middle of the data, they are much less pulled around by a few extreme values.

This matters because of outliers, which are data points that are very different from the rest, such as measurement errors. A single outlier can shift a mean considerably and inflate a variance, which then distorts how every ordinary point is scaled. RobustScaler ignores such points to a much greater degree, which is why it is the preferred choice when the data contains them. Recognizing outliers is therefore part of choosing a scaler.

The key contrast is this. StandardScaler's two statistics, the mean and the variance, are both sensitive to every point, including the extreme ones. RobustScaler's statistics are anchored to the middle of the data, so an unusual point has far less leverage over the transformation.

MinMaxScaler has a different purpose. It shifts the data so that all values of each feature fall inside a specific range, namely between 0 and 1 by default. For a two-dimensional dataset, this means all of the points end up inside a square whose sides run from 0 to 1 along each axis. Unlike StandardScaler, this technique gives you a guaranteed minimum and maximum for each feature.

The mapping is easiest to see by following one value. The original minimum of a feature moves to the bottom of the target range, the original maximum moves to the top, and every value in between lands at a proportional position. The standard calculation for a target range from a to b is shown below, where X_min and X_max are the smallest and largest original values of the feature.

Because the mapping is linear, the ordering of values is preserved. Any value that was larger than another before scaling is still larger after scaling, and the whole feature is compressed or stretched into the chosen interval. This is what gives MinMaxScaler its guaranteed bounds.

Normalizer is different from the first three techniques in one fundamental way: it does not work column by column. It looks at each data point as a whole feature vector and rescales that vector so its Euclidean length is 1. Geometrically, every data point is projected onto the circle of radius 1 in two dimensions, or the sphere of radius 1 in higher dimensions.

The reason the Normalizer must work per point is that its goal is a property of each point, namely its length. A length cannot be fixed one column at a time. Each point is divided by its own length, so every point is scaled by a different number. The divisor is the inverse of that point's length. Points that were far from the origin shrink more, and points already close to the origin are stretched, but in every case the final point has length 1.

This normalization is most useful when only the direction of the data matters and the length of the feature vector does not. Once every point has length 1, comparisons between points depend on their angles rather than on how large their raw values were.