Linear Regression on Continuous Features

Suppose you have a single continuous input feature, such as a value that can take any number within a range, and an output you want to predict from it. You fit two different kinds of regression model to the same training points: a linear regression model and a decision tree regressor. When you draw both predictions over the raw points, the two curves can look very different. This article explains why, and it shows how a change in the feature representation can change what a linear model is able to do.

A linear model is limited by its form. When there is a single input feature, the relationship between input and output is a line. A line has one slope and one intercept. No choice of those two numbers can make the line curve up and down to chase a wavy pattern. So the linear regression curve in a comparison is a straight line, even when the training points clearly bend.

This limitation belongs to the model family, not to the quality of the fit. A linear model that looks poor on wavy data is doing exactly what it was built to do. When you see a straight prediction line cutting through a curved cloud of points, the first explanation to consider is the model's form. Blaming the code or the data comes after that.

A decision tree regressor does not have a fixed functional form like a line. Instead, it builds its prediction by making a sequence of splits on the input feature and assigning an output value to each region those splits create. Because the prediction is built region by region, it can follow bends and irregular patterns that a single straight line cannot. The source describes the tree as able to build a much more complex model of the data.

The source is careful about one qualification. The tree's ability to capture complex patterns depends strongly on how the data is represented. The same tree can look powerful or weak depending on the feature representation it receives. So the real question in a comparison is not only which model is more capable, but how each model consumes the feature it is given.

Binning, also called discretization, splits one continuous feature into multiple features. The source presents it as one way to make a linear model more powerful on continuous data. You choose a set of ranges that cover the input values. Each range becomes a bin. Each data point is then described by which bin it falls into, using a set of indicator features that are 1 for the bin the value falls in and 0 for every other bin.

Notice what this representation does not contain. The original value 5.3 has been replaced by indicator features that record only which range it falls in. A value of 5.9 would produce the same indicator pattern, and so would any other value between 4 and 6. Binning trades fine detail inside each range for the ability to give each range its own weight.

Before binning, the linear model sees one feature, x, and its prediction is a single line. After binning, the linear model sees several indicator features. Each derived feature can carry its own weight. The combined prediction can therefore rise and fall in different regions of the input range. The model is still linear in the features it receives, but the representation of the data has changed, and that change is what makes the fit different.

In the after picture, each bin acts as a separate feature with its own weight. For any input, only one indicator is 1, so the prediction reduces to the weight of the active bin. Across the whole range this produces a set of flat segments, one per bin, instead of one slope across the entire input range. This is a step-shaped fit, similar in spirit to the regions a decision tree creates.

When a model comparison gives a result you did not expect, work through the pipeline in order instead of guessing. Start with the feature representation, then the model family, and only then the fitted curves. The divergence usually appears at one of these three points, and the first mismatch between your expectation and your reasoning tells you where to look.