Scaling Features for Linear Models

Imagine you have followed a familiar recipe for improving a model. You scale the original features, then add interaction and polynomial columns to the training data, and you expect every model to do better. Then you check the Random Forest and its test score has dropped slightly. Nothing crashed, and the arithmetic is not broken. The result simply disagrees with an expectation carried over from linear models.

This article uses that situation as a worked case. You will see how a controlled comparison isolates the effect of the features, how to read a small score change without over-reading it, and why a linear model and a tree-based model respond so differently to the same added columns. The article ends with the single-feature transformations (log, exp, and sin) that also appear in this chapter.

A side-by-side comparison has a fixed order of steps. First, a model is trained on the scaled original features. Second, the same kind of model is trained on a training set that has been expanded with interaction and polynomial columns. Each fitted model is then scored on the matching test set, and finally the two scores are compared. The diagram below traces that order and marks the step where the surprise first shows up.

The unexpected number first appears at the scoring step, not at the training step. Both models were fitted without errors, and the scoring method is the same for both. The surprise becomes visible only when the two test scores are placed next to each other. That is why the comparison step should be read carefully: the question is which change produced the number, and whether it came from the model or from the features.

A comparison is informative only when a single thing changes. In this one, the training and test feature matrices are the only difference between the two runs. The model class, the number of trees, and the scoring method stay the same. If the score moves, the change can be attributed to the features, and the remaining question is how that particular model responds to those particular features.

The source makes a point that is easy to skip: a drop of a small size does not show that interaction features are harmful in general. It shows only that for this model, on this data, the extra columns did not help and slightly hurt. A fair report therefore states both the direction of the change and its size, and avoids turning one small difference into a sweeping conclusion.

A linear model can only combine features additively. With two features, it can represent a contribution from $x_1$ and a separate contribution from $x_2$, and nothing else. It cannot express the idea that the effect of $x_1$ depends on the level of $x_2$. An interaction feature supplies that missing signal as a new column whose value is the product $x_1 x_2$. Once that column exists, the linear model has a new term it can weight, and the combined effect becomes available to it.

The diagram shows the change in the set of terms. Before the expansion, the model has two additive terms. After it, a third term exists that carries the product of the two features. The model's fitted line is still linear in its weights, but the new column lets it respond to a combined effect that the original columns could not express. This is why the source reports that interaction and polynomial features can give a linear model like Ridge a clear boost.

The source states two facts that together explain the pattern. First, even without any added features, the Random Forest already beats Ridge on this data. Second, adding interactions and polynomial terms raises Ridge's performance, while the Random Forest's performance decreases slightly. The two models start from different positions and draw their decisions in different ways, so the same added columns can push their scores in opposite directions.

The lesson is that a feature engineering step has no fixed effect. A Random Forest builds its splits from the features it is given, and the expanded set can make the same signal harder to use well on this data. A linear model, by contrast, gains a new kind of signal it could not use before. The same column is therefore useful to one model and neutral or slightly harmful to the other.

Squared and cubed features are one way to give a linear model curved behavior. A different group of tools applies one mathematical function to one feature at a time. The source names three: log, exp, and sin. Linear models and neural networks are closely tied to the scale and distribution of each feature, so when the relation between a feature and the target is nonlinear, that relation is hard for them to model, especially in regression. Transforming the single feature can make the pattern easier to capture.

The log and exp functions help by adjusting relative scales in the data, so that a linear model or neural network can capture the pattern more easily. The source points to the memory price data from Chapter 2 as an earlier application of this idea. The sin and cos functions suit data that encodes periodic patterns, such as repeating cycles over time, because they map a repeating pattern onto a smooth curve.

Tree-based models depend only on the ordering of a feature's values. A split asks whether a value falls above or below a threshold, so a monotonic transformation that keeps the order intact produces the same partitions of the data. A linear model's fitted line, however, depends on the actual numeric spacing of the values, so the same transformation changes what the line looks like. This is the reason tree-based models are much less affected by how a feature is scaled or distributed.