Feature Scaling for Linear Models

A linear model combines its input features with weights and adds them up. When you give it one input column, the model can only express a straight-line relationship between that column and the target. If the real relationship between the input and the target curves, a straight-line combination of the original column may fit poorly no matter how well the weights are chosen.

One way to give a linear model more flexibility is to add new columns that are powers of the existing ones. A column raised to the power 2, or to the power 3, bends in a way the original column cannot. These new columns are called polynomial features. The linear model still adds its weighted inputs together, but now some of those inputs are curved versions of the original value.

This article follows a code-first path. You will run the scikit-learn code that creates polynomial features, predict what the output will look like, and then use the output to explain how the features are built. The goal is to learn to read the result of the transformation and to find the exact place where your expectation and the real output disagree.

The class PolynomialFeatures is imported from scikit-learn's preprocessing module. You create an object that records the settings you want, call fit on your data, and then call transform to produce the new array. The input here is a single column, so X has shape (3, 1): three samples, one feature each.

Read the output column by column. The first column still holds the original values 2, 3 and 0.5. The second column holds those values squared, and the third holds them cubed. That matches the pattern the source describes: the first output column is the original feature, and the remaining columns are its higher powers, up to the chosen degree. The source example uses a degree of 10 and turns off the constant column in the same way.

The shape tells you how many rows and columns the array has before you look at any individual number. The number of rows does not change. The transformer produces one row per sample, so an input of shape $(n, 1)$ becomes an output of shape $(n, k)$. Only the number of columns changes, and $k$ is set by the degree and by whether the constant column is included.

The degree setting is the highest power that gets generated. With include_bias=False, a single input column produces one feature for each power from 1 up to the degree. The source example, with a degree of 10, produces 10 features for a single column, and the transformed array has one row per sample.

The include_bias setting controls the constant column. When it is turned off, as in the earlier example, the output starts at the original feature. When it is turned on, a constant column is added to the output, and it is placed first in the array in the example below. That constant column adds one to the width of the output for the same degree.

A shape tells you how many columns exist, but not what they mean. The transformer reports the meaning of each column through its feature-name method, which gives the exponent each feature was raised to. The source calls this method get_feature_names; current scikit-learn releases name the method get_feature_names_out. In both cases the output lists the original feature first and then each of its powers.

Reading the names gives you a check against your expectation. The first name should always be the original feature, the first output column should match the input, and the remaining names should step through the powers in order. When include_bias=True, the constant column appears with the name 1, which makes it easy to spot in the list.

Once you see the generated columns, you may notice that their magnitudes are very different. The source points out that the second column reaches values above 20,000 in its example, which is orders of magnitude larger than most of the others. This follows from raising values to powers. When a value is larger than one in absolute value, each higher power multiplies it further. Values between negative one and one shrink toward zero as the power rises.

When a polynomial result looks wrong, check three things in order: the shape, the first column against the original input, and the feature names. The first place where the observed output disagrees with your expectation tells you where the problem is. If any of these checks fail, the problem is in the setup, not in the model.