Scaling Features Before Modeling
Interaction and polynomial features supply that shape directly, which is why Ridge gained so much in the comparison.
Giving a Model More Information
Once a dataset has been prepared, a natural question follows: can we make a model better by giving it more information? One direct way is to create new features from the existing ones. This article looks at two common kinds of new feature, interaction features and polynomial features, and asks a practical question: does adding them actually improve the score on unseen data? The answer depends on which model you use, and seeing that dependence is the main lesson here.
The comparison is run in a fixed way. A model is fitted on the training portion of the data, and then its score is computed on the test portion. The same procedure is run twice for each model: once with the original scaled features, and once with a second feature matrix that includes the added interaction and polynomial terms. Keeping the procedure identical is what makes the two scores comparable. If the procedure changed between runs, any difference in score could come from the change in procedure rather than from the new features.
Why the Same Features Can Help or Hurt
Before any code, hold onto one idea. Whether extra features help depends on what the model can already do with the original features. A model that cannot bend to follow a curve or a joint effect between two inputs may gain a lot from features that carry that shape. A model that already captures those patterns on its own may gain little, and the extra columns can add noise. The comparison in this article is designed to make that difference visible.
Building Interaction and Polynomial Features
An interaction feature combines existing features, often by multiplication, so that a model can respond to their joint effect. If the original features are $x_1$ and $x_2$, a common interaction feature is $x_1 \cdot x_2$.
A polynomial feature raises an existing feature to a power, such as squared or cubed. From a single feature $x$, the squared polynomial feature is $x^{2}$ and the cubed polynomial feature is $x^{3}$.
Notice what both kinds have in common. Neither one introduces new raw measurements. Each new column is computed entirely from columns that already exist. The expanded feature matrix therefore keeps every original column and adds new ones beside them. The diagram below shows which original columns are combined to form each new column and where those new columns sit.
Read the diagram left to right. The first two positions hold the original columns. The three positions after them hold the added columns, in the order they were created. The original columns are still present in the expanded matrix, so the expanded matrix is a superset of the original one.
Expanding one row of two features
A single row has $x_1 = 2$ and $x_2 = 3$. Build the interaction feature and the two polynomial features for this row.
- 1Start with the original values
The row contributes $x_1 = 2$ and $x_2 = 3$ as its original columns.
- 2Build the interaction feature
Multiply the two original columns: $x_1 \cdot x_2 = 2 \cdot 3 = 6$.
- 3Build the squared feature for x1
Square the first column: $x_1^{2} = 2^{2} = 4$.
- 4Build the squared feature for x2
Square the second column: $x_2^{2} = 3^{2} = 9$.
- 5Assemble the expanded row
The expanded row is $[2, 3, 6, 4, 9]$, with the original columns first and the three new columns after them.
The expanded row has five values: two original and three added. The model now receives the joint effect $x_1 \cdot x_2$ as a column of its own.
Tracing the Ridge Fit and Score Lines
The code for this comparison was prepared in earlier steps of the chapter. The variables X_train_scaled, X_test_scaled, X_train_poly, X_test_poly, y_train, and y_test already exist when these lines run. The first pair of lines fits a Ridge model on the scaled original features and reports its test score. The second pair fits a fresh Ridge model on the expanded feature matrix and reports its test score. Trace each line and notice which matrix is passed to fit and which is passed to score.
from sklearn.linear_model import Ridge
# Pair 1: original scaled features
ridge_orig = Ridge()
ridge_orig.fit(X_train_scaled, y_train)
print("Original features:", format(ridge_orig.score(X_test_scaled, y_test), ".3f"))
# Pair 2: expanded features, fresh model
ridge_poly = Ridge()
ridge_poly.fit(X_train_poly, y_train)
print("Expanded features:", format(ridge_poly.score(X_test_poly, y_test), ".3f"))- 1The import brings in the Ridge class, a linear model.
- 2
ridge_origis created as an unfitted Ridge model. - 3fit is called with X_train_scaled and
y_train, so the model learns only from the training portion. - 4score is called with X_test_scaled and
y_test, so the result reflects data the model did not train on. - 5
ridge_polyis a fresh model. It is not the one fitted above, so the two scores come from separate fits. - 6fit is called with X_train_poly, the expanded training matrix, and the same
y_train. - 7score is called with X_test_poly and
y_test, the expanded test matrix and the same test target. - 8The format placeholder with three decimal places only controls how the number is displayed. It does not change the fit.
- fit receives the training inputs and training target only
- score receives the test inputs and test target only
- The second model is a new object, not a reused one
The expanded features do not change the order of the steps. They are created during preparation, passed to the model's fit method, and then used again when score is called on the test data. What changes is only the matrix that flows through the two lines. The target y_train is used for training in both fits, and y_test is used for evaluation in both scores. The source reports that the Ridge test score rose from 0.621 with the original features to 0.753 with the expanded features.
Why Ridge Gains From Expansion
A Ridge model is a linear model. It can only combine features in a weighted sum. It cannot bend to follow a curve or a joint effect between two inputs unless the features themselves already carry that shape. Interaction and polynomial features supply that shape directly, which is why Ridge gained so much in the comparison. The model did not get smarter. It was given columns that already contain the curvature or joint effect it could not produce on its own.
Nonlinear Transformations for Linear Models
Squared and cubed features are one way to help a linear model with a nonlinear relationship. Another approach is to transform a single feature with a mathematical function such as log, exp, sin, or cos. Linear models and neural networks are closely tied to the scale and distribution of each feature. When the relation between a feature and the target is nonlinear, that relation is hard for them to model, particularly in regression. A transformation can reshape the feature so the relation becomes closer to a straight line.
The log and exp functions help by adjusting relative scales in the data, so that the transformed values can be captured better by a linear model or a neural network. As a generated illustration, suppose the target depends on the logarithm of a feature, so that $y = \beta_0 + \beta_1 \log(x)$. A linear model fitted on raw $x$ must bend a straight line through a curve, which it cannot do. Fitted on $\log(x)$ instead, the same relationship is a straight line in the new variable, and the linear model can capture it directly.
The sin and cos functions are useful when the data encodes periodic patterns, such as repeating cycles. Log transformation has also been applied earlier in this course to memory price data, which is a reminder that the same idea recurs across chapters.
Random Forest on the Same Features
What do you think happens?
Before looking at the next model, make a prediction. The Ridge results showed a clear gain from the expanded features. A Random Forest regressor with 100 trees is now fitted in the same two ways, first on the scaled original features and then on the expanded features. What do you expect to happen to its test score when the expanded features are added?
Reveal answer
The score falls slightly, from 0.799 with the original features to 0.763 with the expanded features.
A random forest is built from many decision trees and already captures interactions and nonlinear patterns from the original features without help. It already scored higher than Ridge without any added features, so extra expanded columns add little new information and slightly reduced its performance in this comparison.
| Model | Original features | Expanded features | Direction of change |
|---|---|---|---|
| Ridge (linear) | 0.621 | 0.753 | Improved |
| Random forest (100 trees) | 0.799 | 0.763 | Slightly reduced |
Test scores reported in the source comparison. The same expansion helps one model and slightly hurts the other.
The two rows explain each other. Ridge starts from a low score and gains from features that supply curvature and joint effects. The random forest starts from a higher score, already captures those patterns on its own, and gains nothing new from the extra columns. The source's reported drop for the random forest is small, which is why the paragraph above describes it as slight.
Mistakes When Comparing Feature Sets
Common mistake Reusing one fitted model to score both feature sets
What it looks likeCalling score on the expanded test matrix with a model that was fitted only on the original matrix.
Why it failsThe fitted weights belong to the columns the model was trained on. The expanded matrix has different columns, so the comparison no longer means what it should.
Do this insteadCreate a fresh model for the expanded features and fit it on the expanded training matrix, as the second pair of lines does.
Common mistake Swapping the training and test matrices in fit or score
What it looks likePassing X_test_poly to fit, or y_test to fit as the training target.
Why it failsfit must learn from the training portion. Using test data to train removes the held-out check that score is meant to provide.
Do this insteadfit takes the training inputs and training target. score takes the test inputs and test target.
Common mistake Assuming more features always help
What it looks likeExpecting the random forest to improve just because the Ridge model did.
Why it failsWhether extra columns help depends on what the model can already capture. The source shows the same expansion helping Ridge and slightly hurting the random forest.
Do this insteadCompare each model's two scores under the same procedure before deciding whether to keep the expanded features.
Common mistake Believing the format placeholder changes the result
What it looks likeThinking that writing three decimal places alters how accurately the model was fitted.
Why it failsThe source is explicit that the placeholder controls only the display.
Do this insteadTreat the printed value as a formatted view of the score, and make decisions from the underlying comparison.
Practice Predicting the Outcome
A colleague adds squared and interaction columns to a dataset and fits two models: a Ridge model and a random forest. The Ridge score rises and the random forest score falls slightly. Explain, in your own words, why the same expansion can help one model and hurt the other, and name the two matrices that were passed to fit in the comparison.
Hints
- Think about whether the model can already express curvature or joint effects from the original features.
- The two matrices passed to fit are the original training matrix and the expanded training matrix.
Key Takeaways
- Interaction features combine existing features, often by multiplication, and polynomial features raise a feature to a power such as squared or cubed. Each new column is built from columns that already exist.
- In the source comparison, adding interaction and polynomial features raised the Ridge test score from 0.621 to 0.753, because a linear model gains from columns that carry curvature and joint effects.
- The random forest scored 0.799 on the original features, higher than Ridge, and fell to 0.763 with the expanded features, because it already captures those patterns and the extra columns add little.
- Log, exp, sin, and cos transformations can help linear models and neural networks handle nonlinear relationships and periodic patterns in a single feature.
- Keep the comparison procedure identical across feature sets: fit on the training portion, score on the test portion, and use a fresh model for each feature matrix.