Creating Interaction Features

A model learns from the numbers you give it, so the way those numbers are laid out matters as much as the algorithm that consumes them. One way to enrich a feature representation, especially for linear models, is to add interaction features and polynomial features built from the original data. This kind of feature engineering shows up in statistical modeling and in many practical machine learning applications.

This article follows one concrete case. A linear model is fit to binned data, and the binned model has learned only a constant for each bin. The question is what happens when one more column is placed next to the bin columns. To answer it, you will trace the shape of the feature matrix at each step, because the shapes tell you what the model actually receives.

The source frames the representation as a trace. Each part of the combining expression produces an array, and the final array is the one the linear model receives. Rather than trusting the plot, follow the shape of the data after each construction step. The shape is a pair of numbers: the number of rows, which is the number of data points, and the number of columns, which is the number of features.

In this example the data set has 100 rows, one per data point. Before the stacking step, the original feature and the binned features each have 100 rows. The final printed result for the interaction representation is (100, 20): still 100 rows, and 20 columns. The rest of the explanation shows where those 20 columns come from.

The pipeline above shows the important rule for this section: the row count stays fixed at 100 throughout, and only the column count changes. If the number you print at the end does not match what you count from the two blocks, the problem lies in the stacking step or in the inputs passed to it, not in the regression. Checking shapes after each step lets you find that point quickly.

The new data set has 20 features, and they come in two families. The first family records which bin each data point falls into. These are the bin indicator columns: for each bin, a column that marks whether the data point lies inside that bin. The second family takes the original x-axis feature and restricts it to each bin. Each product feature holds the original x-axis value inside its bin and zero everywhere else.

Both families are concatenated into one matrix, so the linear model sees a single set of columns and fits them together. Each bin contributes one indicator column and one product column, so the bin ranges are mirrored in both halves of the matrix. Reading a column position therefore tells you which bin it belongs to and whether it is an indicator or a restricted copy of the original value.

The code above uses placeholder zero arrays only to show the shapes; in the real representation the blocks hold the indicator and product values. The shape reasoning is what carries over: two blocks of 10 columns each, stacked horizontally over the same 100 rows, produce 20 columns.

Start with what the binned model already does. A linear model on binned data learns a constant value for each bin. Each bin is an interval of the original input, and every point that falls inside it receives the same predicted value. The result is a step-like shape: flat within each bin, with jumps at the bin edges. Such a model cannot express a slope on its own, because a constant has no slope.

When the linear regression is fit on the binned-plus-product representation, the result is a separate line for each bin. The lines do not have to share one slope. Each bin ends up with its own offset and its own slope. The model is still linear in its features, but the features now let the slope change from one bin to the next. In symbols, within bin $j$ the prediction takes the form $a_j + b_j x$, where $a_j$ is the offset and $b_j$ is the slope for that bin.

A linear model can learn more than offsets. It can also learn slopes. One way to give the model a slope is to add the original feature back in, next to the binned columns. The original feature and the binned features are two blocks of columns describing the same 100 data points. The stacking step places them side by side, so each row carries the raw value together with its bin indicators.

The model now has a single coefficient that applies to the raw input value across all bins. That shared coefficient acts as one global slope, while the bin columns continue to supply a separate level for each bin. In the source example this produces an 11-dimensional data set: the original feature plus the 10 bin columns.

Notice that the stacking step adds columns, not rows. Adding columns changes what each data point looks like to the model, while the number of data points stays the same. That is why the printed row count remains 100 after stacking, and why the column count is always the sum of the column counts of the blocks being joined.