Linear Regression with Binned Features
Interaction and polynomial features enrich a feature representation, especially for linear models.
Why Binned Data Produces Steps
Interaction and polynomial features enrich a feature representation, and they are especially useful for linear models. This article focuses on one case: a linear model fit to binned data. A model learns from the numbers you give it, so the way those numbers are laid out matters as much as the algorithm that consumes them.
Binning turns a continuous input into intervals. Each bin is an interval of the original input value, and every point that falls inside it is represented by the bin it belongs to. A linear model fit to these binned columns learns a constant value for each bin. Every point inside a bin therefore receives the same predicted value.
What do you think happens?
If a linear model learns only one constant per bin and you plot its predictions against the original input, what shape do you expect to see?
Reveal answer
Flat segments within each bin, with jumps at the bin edges.
Because the model learns one value per bin, predictions stay flat across each bin's interval. The change happens only when the input crosses from one bin into the next, which produces a step-like shape.
Tracing What the Binned Model Predicts
Trace the binned-only model across the input range. Start at the lowest bin. Every input value inside that interval gets the same prediction. Move right until the input crosses the first bin edge. The prediction then jumps to the constant learned for the next bin, and it stays there until the next edge. The model has no way to say that predictions should rise gradually inside a bin, because it has learned no slope.
Adding the Original Feature Back In
A linear model can learn more than offsets; it can also learn slopes. One way to give it a slope is to add the original feature back in, placed next to the binned columns. The model now has a single coefficient that applies to the raw input value across all bins. That shared coefficient acts as one global slope. The bin columns continue to supply a separate level for each bin.
Put the two parts together. The prediction within any bin is the level the bin columns supply, plus the global slope applied to the raw value. Moving the input within a single bin changes the prediction only through the slope term. Crossing a bin boundary changes the prediction through the bin columns, which switch to a different level.
Stacking Two Blocks Column by Column
The combination step is a single operation. The original feature and the binned features are two blocks of columns describing the same 100 data points. A horizontal stack places them side by side, so each row carries the raw value and its bin indicators together. The model then treats every column as one input dimension when it fits. The stacking step adds columns and leaves the number of rows unchanged.
(100, 1) (100, 10) (100, 11)- X is the original feature with one column and 100 rows
- X_binned has one indicator column per bin, here 10 bins
np.hstackadds the columns side by side and keeps the 100 rows
Predicting the shape after stacking
The original feature has 100 rows and 1 column. The binned block has 100 rows and 10 columns. What is the shape of the stacked matrix?
- 1Check the row counts
Both blocks have 100 rows, so the stack is valid and the row count stays at 100.
- 2Add the column counts
The original block contributes 1 column and the binned block contributes 10, so the total is 1 + 10 = 11.
- 3Compare with the source
The source example describes an 11-dimensional dataset, which matches this count.
The stacked matrix has shape (100, 11).
Where the Column Count Surprises You
Common mistake Expecting the row count to change after stacking
Why it failsStacking adds columns, not rows. Each data point stays the same; it simply carries more input values.
Do this insteadRead the row count first. After a horizontal stack it should match the row count of each input block.
Common mistake Counting only the bin columns and forgetting the original feature
Why it failsThe combined matrix contains the sum of both blocks' columns. Leaving out the original column gives the binned-only model, which has no slope.
Do this insteadCount one column for the original feature plus the number of bin columns.
Common mistake Assuming the model is wrong when the column count does not match a hand count
Why it failsIf the printed column count differs from your count of the two blocks, the stacking inputs are the likely cause rather than the regression itself.
Do this insteadPrint the shape of each block before the stack and then the shape of the result, and compare them step by step.
Practice: Predict the Column Count
A dataset has 100 rows. You build a binned block with 5 bins, one indicator column per bin, and stack the original single feature in front of it. Before running anything, predict the shape of the combined matrix, then say which column is the global slope term and which columns supply per-bin levels.
Hints
- Count the columns contributed by each block separately before adding them.
- The row count should be the same as the inputs.
Suppose a colleague stacks the binned block with the original feature, but the printed matrix has 100 rows and 5 columns. Describe where in the combination step the problem most likely lies, and what shape check you would run first to find it.
Hints
- Compare the column count with the sum of the two blocks.
- Check the inputs passed to the stack before questioning the regression.
Key Takeaways
- A linear model fit to binned data learns one constant per bin, so its predictions are flat within each bin and jump at the bin edges.
- Adding the original feature as an extra column gives the model one shared coefficient, a global slope, while the bin columns still supply a level for each bin.
- Stacking two blocks side by side adds their column counts and keeps the row count the same; for 100 points with one original column and 10 bin columns, the result is 11 columns.
- When a combined model diverges from expectations, check the shapes of the inputs and the stacked result first, before suspecting the regression itself.