Polynomial Features and Interactions

A single x-axis feature gives a binned linear model only one shared slope, even when each bin has its own offset.

Programming

Why One Slope Is Not Enough

Suppose you want a linear model to predict a target from a single x-axis value, but the relationship between x and the target is not the same across the whole range of x. Maybe the target rises steeply for small x values and then falls for larger ones. A straight line has only one slope, so it struggles with this kind of data. This article looks at one common response: split x into bins, give each bin its own offset, and then add a feature that lets each bin also have its own slope.

What do you think happens?

If you split x into bins and give each bin its own offset, can a linear model with one x feature follow data where each bin has a different rate of change?

Yes, separate offsets are enough to fix the slope problem
No, the model still applies one shared slope to every bin
Only if there are more than ten bins

Reveal answer

Answer:

No, the model still applies one shared slope to every bin

Per-bin offsets let the model shift its prediction up or down from one bin to the next, which helps. But the model still has only one x-axis feature, and that feature has only one slope, so the same downward slope is applied in every bin. Separate offsets do not change the rate of change.

Offsets Versus Slopes in Binned Models

Binning turns a numeric feature into a set of indicator features. Each indicator records whether a data point falls in a particular bin. A point in bin 0 has indicator 1 for bin 0 and 0 for every other bin. Once the input is in this form, the model can learn a separate weight for each indicator. That indicator weight acts as the offset for its bin, which means it moves that bin's prediction up or down.

The offset is only half of what a bin can need. Starting from a model that has already been given binned input, each bin receives its own offset, so predictions can jump from one bin to the next. However, the model still has only one x-axis feature, and that feature has only one slope. The same slope is applied to every bin. That is the limitation: the model can say how the output changes with x in one way, everywhere.

Can each bin follow its own trend? With binned input and one x feature, both bins share one slope. With product features added, each bin has its own offset and its own slope.

Where the Shared Slope Breaks Down

A shared slope means the model can only describe how the output changes with x in one way, everywhere. When the true relationship has a different rate of change in each bin, one slope cannot follow it. Adding a separate offset per bin helps with the jumps between bins, but the slope problem remains. Each bin's trend is still being forced into the same slope.

The fix is to give each bin its own slope as well as its own offset. The method used is an interaction feature: a feature that combines two other features so the model can respond to their combination. Here the two features are the bin indicator, which records which bin a data point is in, and the original x-axis value. Their product is the feature that gives each bin its own slope.

Building a Product Feature

A product feature is the original x value within its bin and zero everywhere else. For each bin, you multiply x by that bin's indicator. A point inside the bin keeps its x value, because the indicator is 1. A point outside the bin becomes 0, because the indicator is 0. Because of this zero-outside-the-bin behavior, the linear model can only use a product feature to adjust predictions for points in that bin. For points in other bins, the product feature contributes nothing.

How do the bin indicator and x combine row by row? For the point x = 2.5 in bin 1, the indicators mark bin 1, and the product for bin 1 keeps x while the product for bin 0 becomes zero.

Tracing Four Points Through the Product Columns

The source's dataset is large, so its numpy construction hides the individual values. The following example uses plain Python and only four data points with two bins. Bin 0 holds points with x below 2.0, and bin 1 holds points with x at or above 2.0. The construction is the same as the full version: one indicator per bin, and one product of x with each indicator. Here the points are 0.5, 1.5, 2.5, and 3.5, which are generated for this example.

python

xs = [0.5, 1.5, 2.5, 3.5]

def bin_of(x):
    return 0 if x < 2.0 else 1

N_BINS = 2
rows = []
for x in xs:
    indicators = [1 if bin_of(x) == b else 0 for b in range(N_BINS)]
    products = [x * ind for ind in indicators]
    rows.append(indicators + products)

for row in rows:
    print(row)

xs = [0.5, 1.5, 2.5, 3.5]

def bin_of(x):
    return 0 if x < 2.0 else 1

N_BINS = 2
rows = []
for x in xs:
    indicators = [1 if bin_of(x) == b else 0 for b in range(N_BINS)]
    products = [x * ind for ind in indicators]
    rows.append(indicators + products)

for row in rows:
    print(row)

Output (expected)

[1, 0, 0.5, 0.0]
[1, 0, 1.5, 0.0]
[0, 1, 0.0, 2.5]
[0, 1, 0.0, 3.5]

Notes on the code

bin_of decides which bin a point belongs to
indicators holds one 0 or 1 per bin
products multiplies x by each indicator, so only the point's own bin keeps x
each row is the indicators followed by the products

Worked example

Reading the row for x = 2.5

Find the full feature row for the point x = 2.5 when bin 0 covers x below 2.0 and bin 1 covers x at or above 2.0.

1
Find the bin
2.5 is at or above 2.0, so the point belongs to bin 1.
2
Write the indicator columns
The bin 0 indicator is 0 and the bin 1 indicator is 1, giving [0, 1].
3
Compute the first product column
Multiply x by the bin 0 indicator: 2.5 times 0 gives 0.0, because the point is outside bin 0.
4
Compute the second product column
Multiply x by the bin 1 indicator: 2.5 times 1 gives 2.5, because the point is inside bin 1.
5
Assemble the row
The complete row is [0, 1, 0.0, 2.5]. The product column for bin 1 is the only nonzero product.

Result

The row for x = 2.5 is [0, 1, 0.0, 2.5]. Its indicators mark it as bin 1, and only the bin 1 product carries the x value.

Which feature column switches on for which bin? The bin 1 indicator is on, and the bin 1 product carries x, while the bin 0 columns are zero for this point.

From Four Points to the Full Dataset

The source builds all of the interaction features with one line. It multiplies the single x column by the bin indicator columns, producing one product column per bin, and then joins the indicator columns and the product columns side by side. The same idea you traced by hand for four points is done in one step for every row.

python

import numpy as np

# X_binned: one indicator column per bin
# X: the single original x-axis column
X_interaction = np.hstack([X_binned, X * X_binned])

import numpy as np

# X_binned: one indicator column per bin
# X: the single original x-axis column
X_interaction = np.hstack([X_binned, X * X_binned])

Notes on the code

X_binned holds the bin indicator columns
X * X_binned multiplies the x column against each indicator column, one product column per bin
np.hstack joins the indicator columns and the product columns side by side

In the source dataset there are 10 bins. That gives 10 indicator features and 10 product features, for a total of 20 features across 100 data points. The number of columns is therefore twice the number of bins, which is why the shape is (100, 20). Each row still has exactly one nonzero indicator, and its product columns are zero everywhere except in its own bin.

Reading the Coefficients of Interaction Features

Once the model is fitted on the product representation, every bin has two learned weights: one on its indicator and one on its product. The indicator weight is the bin's offset, and the product weight is the bin's slope. Because a product column is zero outside its bin, the other bins' weights do not affect a point that sits in a different bin.

Formula, step by step

1
b_0 and b_1 are the bin indicators, each 0 or 1
2
o_0 and o_1 are the indicator weights, which act as each bin's offset
3
s_0 and s_1 are the product weights, which act as each bin's slope
4
x times b_0 is nonzero only for points in bin 0, so s_0 applies only there
5
For a point in bin 0, the prediction reduces to o_0 plus s_0 times x
6
For a point in bin 1, the prediction reduces to o_1 plus s_1 times x

Read the coefficients this way: the offset tells you where a bin's line starts, and the slope tells you how that line rises or falls with x. Two bins with different slope weights follow different trends, which is exactly what one shared slope could not do.

Mistakes When Reading Bin Features

Common mistake
Thinking per-bin offsets give each bin its own slope
What it looks like
Adding only the bin indicators to a model with one x feature and expecting bins to have different trends.
Why it fails
Indicators shift the line up or down, but the model still has one x feature with one slope.
Do this instead
Add product features so each bin gets a slope weight as well as an offset weight.
Common mistake
Treating the indicator weight as the slope
What it looks like
Reading the weight on the bin 1 indicator as how steeply bin 1 rises.
Why it fails
The indicator weight shifts the bin's prediction; the slope is carried by the product weight.
Do this instead
Read the indicator weight as the offset and the product weight as the slope.
Common mistake
Expecting a product column to be nonzero for every point
What it looks like
Assuming the bin 0 product for x = 2.5 is 2.5.
Why it fails
The product is the original x value only inside its bin and zero everywhere else.
Do this instead
For x = 2.5 in bin 1, the bin 0 product is 0.0 and the bin 1 product is 2.5.
Common mistake
Forgetting that each bin adds two columns, so 10 bins give 20 features rather than 10

Practice: Product Features for Two New Bins

MEDIUM

Using the same two-bin rule as the worked example (bin 0 for x below 2.0, bin 1 for x at or above 2.0), write the full feature row [bin 0 indicator, bin 1 indicator, x times bin 0, x times bin 1] for the points x = 1.0 and x = 3.0. Then state which bin's slope weight would affect each point.

Hints

Decide the bin first by comparing x with 2.0.
Set the indicator for the point's bin to 1 and the other to 0.
Multiply x by each indicator. Only the point's own bin keeps x, and the other product is 0.0.

Your attempt (not saved)