Polynomial Features and Model Complexity

Imagine you take the same dataset and make one change to its inputs: you add interaction and polynomial columns built from the original features. You then fit two models on the two versions of the data. One model is Ridge, a linear model. The other is a Random Forest. Before you read on, decide what you expect to happen to each model's test score.

If your prediction was that every model improves when you add features, you are not alone. That expectation is common, and this comparison is built to test it. The point of the exercise is not that feature engineering is harmful. It is that a feature engineering step is not a universal upgrade, and its effect depends on the model that consumes the features.

Each row of a feature matrix is one example, and each column is one input the model sees. Interaction features and polynomial features add new columns that are computed from the existing ones. Take a simple case with two original features, $x_1$ and $x_2$. Adding an interaction term contributes the product $x_1 \cdot x_2$. Adding polynomial terms contributes squared versions such as $x_1^{2}$ and $x_2^{2}$. The original columns stay in place; the new columns sit beside them.

Read the diagram column by column. The two original columns survive unchanged. The three new columns each come from a specific pair or single original feature: the interaction column comes from both $x_1$ and $x_2$ together, while the squared columns each come from one feature alone. The model now receives five inputs where it previously received two, and the scoring and training steps still work on this wider matrix in exactly the same way.

A linear model like Ridge makes its prediction by adding up its inputs, each multiplied by a weight. On its own, it can only combine features additively. It cannot, by itself, represent the idea that the effect of $x_1$ depends on the value of $x_2$. An interaction column gives it exactly that ability: the product $x_1 \cdot x_2$ is a new input whose weight can capture a joint effect. Squared columns play a similar role by giving the linear model a curved term to work with.

The key idea is that the added columns supply a new kind of signal. The linear model was not able to express that signal from the original features, so Ridge's score can rise when the expanded columns are included.

A Random Forest does not combine features additively. It builds its predictions from many trees, and each tree splits the data by asking questions about individual features. The source notes that the Random Forest already beat Ridge on the original features before any expansion. When the expanded set is added, the forest has more candidate splits to consider, and the same underlying signal can become harder to use well on this particular data. The result in this comparison is a slight decrease, not a collapse.

Before reasoning about why a model behaves the way it does, start with the measurement. The comparison has four stages: preparing the data, expanding the features, fitting the model, and scoring it on the matching test set. The data is scaled before the fit, and the expanded training and test arrays are created from the same original features. Each fit is scored on the test set that matches its own feature set.

Work through the flowchart step by step. Data preparation and feature expansion are where the two runs begin to differ, but neither step produces a number to be surprised by. Fitting uses the same model class, the same number of trees, and the same scoring method in both runs, so it is not the source of the change. The unexpected number first appears when each fitted model is scored on its matching test set. That is the step where a learner who expected every model to improve first sees the Random Forest result as a failure.

Once both fits are scored, you have two numbers for the same model type: one for the original features and one for the expanded features. The score belongs to the feature set used in that run, so label each one before you compare them. The difference between them is the effect of the added features, and only that effect, because nothing else changed between the runs. Its sign tells you the direction, and its magnitude tells you the size.

Squared and cubed features are one way to give a linear model curved behavior. A second group of tools applies a single mathematical function to one feature at a time. The source names three: log, exp, and sin. Linear models and neural networks are closely tied to the scale and distribution of each feature, so when the relation between a feature and the target is nonlinear, that relation is hard for them to model, especially in regression. A transformation can reshape the feature so the pattern becomes easier to fit.

The source points to the memory price data from Chapter 2 as an earlier application of the log and exp idea. Tree-based models only depend on the ordering of feature values, so they are much less affected by how a feature is scaled or distributed. This is one reason the same transformation that helps a neural network can make little difference to a Random Forest.