Impact of Binning on Linear Models and Decision Trees
Binning turns a continuous feature into bins, and one-hot encoding turns each bin into its own indicator column.
Predicting the Width Before Running Anything
This article starts with a small program rather than a definition. The program builds one continuous feature, splits it into three bins, converts each row's bin membership into indicator columns, and then appends the original feature back on. Before you look at any printed output, try to predict how many columns the final feature matrix has. A common first guess is three, because that counts only the bin columns. The answer depends on keeping track of both parts of the matrix, and the rest of the article uses the printed shapes as evidence for each step.
What do you think happens?
The feature is split into three bins and the original continuous column is appended after the one-hot bin columns. How many columns does the final feature matrix have?
Reveal answer
4
Three bins produce three one-hot indicator columns, and appending the original feature adds one more column, giving 3 + 1 = 4. Guessing 3 is the common mistake, because it counts the bin columns and forgets the original feature.
From Continuous Values to Bin Numbers
Binning turns a continuous feature into a categorical one by dividing its range into intervals called bins. Every value that falls inside the same interval is treated as the same thing. In the program, two edges, -1.0 and 1.0, cut the number line into three intervals: everything below -1.0, everything from -1.0 up to 1.0, and everything at or above 1.0. The function np.digitize reports which interval each value belongs to, numbering the intervals from zero.
[0 1 1 1 2]- Five raw feature values, stored as a column of shape (5, 1)
- Two edges create three intervals, so the valid bin numbers are 0, 1, and 2
np.digitizereturns one bin number per value, not one per interval
Trace the value 0.2 through the diagram. It is greater than -1.0 and less than 1.0, so np.digitize assigns it bin number 1. Its indicator row then has a 1 in the column for bin 1 and 0 in every other column. The value 0.9 also lands in bin 1, so it produces exactly the same indicator row. The value 1.4 lands in bin 2, and the value -1.5 lands in bin 0.
One-Hot Indicator Columns
A bin number alone is not a good input to a linear model. If the model received the bin numbers 0, 1, and 2 as one column, it would multiply that column by a single weight. That forces the bins to lie on a straight line with equal spacing, which is not what binning is meant to achieve. The next step converts each bin number into its own column. This conversion is one-hot encoding: each row contains a 1 in the column for its bin and 0 everywhere else.
(5, 3)
[[1. 0. 0.]
[0. 1. 0.]
[0. 1. 0.]
[0. 1. 0.]
[0. 0. 1.]]- The bin numbers are reshaped to a column so each row is one sample
- Each of the three bins gets its own indicator column
- Rows 2, 3, and 4 are identical because their values all fall in bin 1
Tracing Shapes Through the Pipeline
The binned-feature pipeline has four stages, and each stage produces an array whose shape you can predict. The raw feature is a column of n samples. The bin index is one number per sample. The one-hot matrix has one column per bin. The concatenated matrix places the original feature next to the indicators. Checking each shape in turn is how you find the first stage that disagrees with your prediction.
features = indicators # shape (5, 3)Three indicator columns only, which is the common first guess
features = np.hstack([indicators, x]) # shape (5, 4)Three indicator columns plus the original feature, giving four columns
Predicting and verifying the final matrix shape
Five raw feature values pass through the three-bin program described above. What is the shape of the final feature matrix, and at which stage would a mistake first show up?
- 1Start with the raw feature
The five values are stored as a column of shape (5, 1), so n = 5 samples and one original column.
- 2Compute the bin index
np.digitizewith edges -1.0 and 1.0 returns one bin number per sample, giving a vector of shape (5,) with values 0, 1, 1, 1, 2. - 3Apply one-hot encoding
There are three bins, so each row gets three indicator columns. The one-hot matrix has shape (5, 3).
- 4Append the original feature
Adding the single original column next to the three indicators gives a final matrix of shape (5, 4). The width is 3 + 1 = 4, not 3.
- 5Locate the first possible failure
If the final matrix has width 3, the original column was dropped at the concatenation stage. If the bin index has shape (5, 1) instead of (5,), the one-hot encoder may receive the wrong layout. Each check points to one stage.
The final matrix has shape (5, 4): five rows and four columns, three bin indicators plus the original feature.
Why the Linear Model Gains Flexibility
Before binning, a linear model can express only a single trend across the whole range of the feature. After binning, it has a separate weight for each indicator column, so it can assign a different value to each bin. Each bin's weight acts as that bin's constant prediction. The prediction therefore becomes a set of horizontal steps, one per bin, with the bin boundaries marking where one step ends and the next begins.
The source notes that the fitted linear model and the decision tree both appear as horizontal steps on the same dataset, because both predict one constant per bin. The shape of the prediction is now set by the bins rather than by the original continuous feature. This is the central observation of the experiment.
| Model | Before binning | After binning | Net effect |
|---|---|---|---|
| Linear model | One trend across the whole range | A separate weight per bin, so a different value per bin | Large gain in flexibility |
| Decision tree | Chooses its own split points on the continuous feature | Split points are already free to fall anywhere useful | Little or no benefit |
How each model family responds to binning, and why the reasons differ
Why Adding the Original Feature Matters
A linear model with binned inputs learns one constant per bin. That means it can express an offset for each bin, but not a slope within a bin, because the indicators carry no information about where a point sits inside its interval. Linear models can learn slopes in general, so the missing piece is an input that carries the continuous value. Adding the original feature back into the matrix supplies that input. The indicators set the level for each bin, and the original column lets the model tilt the prediction within a bin.
The source gives a concrete count for this idea: ten bin indicator columns plus the original column gives eleven columns in total, which the source describes as an 11-dimensional dataset. This is a different configuration from the three-bin program above, but the arithmetic is the same. The total width is always the number of bins plus one when the original feature is kept.
Mistakes Beginners Make with Binned Features
Common mistake Counting only the bin columns when predicting the width
What it looks likePredicting 3 columns for a three-bin feature
Why it failsThe original continuous column is still in the matrix once it has been appended.
Do this insteadAdd one for the original feature: three bins plus one original column gives four.
Common mistake Feeding the bin numbers 0, 1, and 2 as a single column
What it looks likePassing the raw output of np.digitize straight into a linear model
Why it failsOne weight on one column forces the bins onto a straight line with equal spacing, which removes the benefit of binning.
Do this insteadConvert the bin numbers into one-hot indicator columns before fitting the linear model.
Common mistake Expecting a decision tree to improve after binning
What it looks likeBinning a feature in advance and assuming the tree will split it more accurately
Why it failsA tree already chooses its own split points on the continuous feature, so binning gives it nothing new in that respect.
Do this insteadUse binning for linear models. Do not expect a tree-based model to gain from it in general.
Common mistake Waiting for the model's accuracy to reveal a pipeline bug
What it looks likeNoticing a poor score and only then investigating the feature matrix
Why it failsA shape error appears at the shape check long before it shows up in a score, so the score hides where the problem started.
Do this insteadPrint the shape after each transformation and compare it with your prediction. The first unexpected shape marks where the divergence happened.
Practice: Predict Before You Print
A continuous feature is split into four bins using three edges. The one-hot indicator matrix is then concatenated with the original continuous column. Predict the shape of the final feature matrix for n samples, then state the stage where a shape of (n, 4) would reveal a missing original column.
Hints
- Four bins produce four indicator columns, not three, and the edge count is one less than the bin count.
- The original continuous column adds exactly one more column after the indicators are appended.
- A final width of 4 would mean the original column was dropped, since the correct width is 5.
Key Takeaways
- Binning divides a continuous feature's range into intervals, and
np.digitizeassigns each value a bin number starting from zero. - One-hot encoding turns each bin number into its own indicator column, so a model sees a separate input per bin rather than one ordered number.
- Adding the original continuous feature back gives the matrix one extra column, so the width is the number of bins plus one, which lets a linear model learn a slope.
- Binning gives a linear model much more flexibility by letting each bin have its own weight, while a decision tree gains little or nothing because it already chooses its own split points.
- Check the array shape after each transformation. The first stage whose shape differs from your prediction is where the pipeline first went wrong.