Linear Models vs Tree-Based Models

A dataset does not have one best representation in isolation. The right way to represent it depends on the meaning of the data and also on the kind of model you plan to train. This article compares two large, widely used families of models: linear models, and tree-based models such as decision trees, gradient boosted trees, and random forests. Their behavior differs a great deal once you change how the input features are represented.

To make the difference concrete, we use the wave regression dataset from Chapter 2. It has only one input feature, which makes it a clean test case. With one feature, you can draw the model's prediction as a curve over the input axis and see exactly what the model is able to express. The wave points rise and fall, so the underlying relationship is not a single straight trend.

A linear model is limited in a specific way: it can only model linear relationships. With a single input feature, a linear relationship is a line. The model's prediction is a weighted sum of the input plus an intercept, so no amount of fitting can make its curve bend. The best it can do is choose the slope and intercept that place one straight line as close to the points as possible.

This is the core reason a linear model struggles with the wave data. The points follow a shape that a single line cannot match. Training can change the slope and the intercept, but the output will always remain one straight line over the input range.

The comparison below summarizes what you should see when both models are fit to the same wave data and their predictions are drawn over a dense grid of input values together with the original points. Both fits use the same arrays, so any difference between the two curves comes from the model, not from the data.

When you look at the figure, ask which curve can follow the bends in the data. The linear model's curve is a single straight shape across the whole input range. The decision tree's curve is built from regions created by its tests on the feature, so it can take a more complex shape.

A decision tree can build a much more complex model of the data than a line. The tree does not rely on a single global formula. Instead, it partitions the input range using tests on the feature and makes predictions from the regions those tests create. Each test is a threshold on the single input. Splitting the range again inside one side produces nested regions, and each final region gets its own prediction.

The number of divisions depends on how the tree grows. The argument min_samples_split=3 in the source code sets the minimum number of samples a node must contain before the tree may split it further. Smaller values let the tree create more divisions, which generally means a more detailed curve, while larger values keep the tree simpler. This setting is one of the ways the tree's flexibility is controlled.

The source describes one concrete way to make a linear model more powerful on continuous data: binning, which is also known as discretization. Binning takes one continuous feature and splits it into multiple features, each covering a different part of the input range. Once the feature is split this way, the linear model no longer has to fit one line across the whole input. It receives several separate inputs and can assign each one its own weight.

The diagram shows the change in representation. Before binning, the model sees one column whose values can be any number in the feature's range. After binning, the same information is expressed as several columns, one per bin. Each column records whether the input falls inside that bin's range. This indicator description of each bin is the standard way binned features are read, and it is how the new columns are meant to be interpreted here.

Once the input is represented by indicator columns, the linear model still computes a weighted sum, but now each bin has its own weight. For any input, exactly one indicator is on, so the prediction is the weight of the bin that input falls into. Moving from one bin to the next changes the output by a jump determined by the two weights. The overall curve becomes a staircase rather than a single slope.

Suppose a linear model's curve does not match what you expected from the points. The fastest way to find the problem is to ask three questions in order. Each question checks a different part of the pipeline from raw feature to prediction.