Feature Scaling and Normalization

Many real datasets contain a column whose values are mostly small, with a handful of extremely large values sitting far to the right. Counts, prices, and durations often look like this. Such a column can carry useful information about the target, yet a linear model can still struggle with it.

Before reading on, make a prediction. Suppose you fit a ridge regression on these raw columns. Will the fit be good, or will the very large values get in the way? The fastest way to answer is to run the model on the raw data and look at the score on the test set. The rest of this article explains what you are likely to see and why, and how a transformation can change the result.

A linear model fits a straight-line relationship between each feature and the target. When a few values are enormous, they dominate the fit. The many small values are squeezed into a tiny region of the scale, where the model cannot tell them apart well.

Picture a feature whose values cluster near zero while a few sit far above the rest. Plotted, it shows a long tail stretching to the right. Statisticians call this a skewed distribution. The source describes this pattern as many small values together with a few very large ones, and notes that it is very common in practice.

A transformation that compresses large values makes the feature easier for a straight-line relationship to describe. A logarithm is one such transformation. After it is applied, the very large values move much closer to the bulk of the data, while the small values, which were crowded together, occupy a wider stretch of the scale. The feature becomes more evenly spread.

The transformation changes the scale of the feature rather than discarding any of its values. Every row keeps its value; only the distances between values change.

In the diagram, the raw values run from 0 to 2000, a gap of 2000 units between the smallest and largest. After the transformation the same five values run from 0.00 to 7.60. The large value no longer towers over the rest, and the small values are spread more evenly across the scale.

The form log(x + 1) is used because many count values are zero, and the logarithm of zero is undefined. Adding 1 moves every value of zero or more into a range where the logarithm is defined. In Python this is usually written with NumPy's np.log1p, which computes the logarithm of one plus the input directly.

There is no single transformation that always works. The source notes that finding the best transformation for each combination of dataset and model is somewhat of an art. Usually only a subset of the features should be transformed, and sometimes each feature needs its own transformation.

The model family matters. Tree-based models are generally unaffected by these transforms, while linear models often depend on them. Binning, polynomials, and interactions can also have a large influence on less complex models such as linear models and naive Bayes models. Other models, including SVMs, nearest neighbors, and neural networks, may sometimes benefit, but the effect is usually less predictable.

Read the flowchart as a set of questions rather than a fixed rule. The first check is the model family, because tree-based models usually do not need this step. For a linear model, the second check is whether the feature is skewed in the way described above. The flowchart is a practical guide; the source is explicit that the choice of which features to transform is an empirical one, so the test score remains the final judge.

Transformations change the values of existing features. A second lever changes how many features the model sees. The source warns that it is tempting to create many new features, but adding features makes all models more complex and raises the chance of overfitting. When the feature count is high, it can be wise to keep only the most useful features and discard the rest, which can produce simpler models that generalize better.

The source describes three basic strategies for feature selection: univariate statistics, model-based selection, and iterative selection. All three are supervised, meaning they use the target. This article focuses on univariate selection, where each feature is scored on its own against the target and only the top-scoring share is kept. SelectPercentile implements this by keeping a chosen percentage of the features.

After selection, compare the held-out test score with and without the reduction. The source's point is that a simpler model may generalize better, so a score that holds steady or improves on the test set with fewer features is evidence that the reduction helped. A drop in test score means the discarded features contained useful information.