Regression With Tree-Based Models

Suppose you have a table of past observations and you want a program to predict something about a new case. One family of methods for doing this is the decision tree. A decision tree makes its prediction by asking a sequence of questions about the input values, and each answer narrows down what the prediction should be. This article follows one idea through the lesson: the same tree can handle categories and numbers, its output is not a simple ordered function of a feature, and it has a limit you must keep in mind when predicting numbers.

A decision tree is built from questions about feature values. Each question has a yes or no style answer, such as whether a value is below a cutoff. Starting at the top of the tree, the program follows one branch according to the answer, asks the next question on that branch, and continues until it reaches an end point. The end point is where the tree gives its final answer for that input.

Think of the tree as a set of nested cutoffs. The first question divides all possible inputs into two groups. Each later question divides one of those groups again. Because of this nesting, every input ends up in exactly one end point, and each end point holds one answer. The rest of the article looks at what those answers look like when the tree is used to predict a number.

The source introduces decision trees in the context of classification first. In classification, each input is assigned to a class by following a path of questions about feature values. Then it explains that everything said about classification also applies to regression. Keeping the two tasks separate in your head makes the rest of the material easier to follow.

The difference between the two tasks is in what the final answer is. A classification tree ends in a class label. A regression tree ends in a numeric prediction. You can think of a regression tree as answering a sequence of questions about the input and then returning a number that corresponds to where the input landed. The questions and the way the tree is grown are the same idea in both cases; only the value stored at the end point changes.

Now trace what happens when you slowly increase one feature and watch the prediction. The tree does not produce a smooth curve. It asks its cutoff questions in order, so as the feature moves from one range into the next, the answer stays the same across an entire range and then jumps to a different value when a cutoff is crossed. The output is piecewise constant: flat over each region, with steps between regions.

Read the diagram from left to right as the feature grows. Inside the band from 2 to 5, every input gets the same prediction, 14, no matter where it sits in that band. The prediction only changes when the input crosses a cutoff. This is why the output does not move in one direction: it goes from 10 up to 14, down to 9, and up to 20 as the feature increases. The numbers here are illustrative, chosen to show the shape a tree can produce, not values from a specific model.

The source notes that classification tree outputs are not monotonous, which means you cannot summarize them by a simple ordering of a feature. In plain terms, as a feature increases, the predicted class does not have to move steadily in one direction. Each path from the top of the tree to a final answer is a combination of conditions, and different paths can lead to the same class or to different classes.

Imagine a tree whose first question separates the lowest values of a feature into one class. Its second question applies only to values above that first cutoff, and it sends those values into two different outcomes depending on where they fall. The class does not move in one direction as the feature grows: low values belong to the first class, the middle values can belong to a second class, and the highest values can return to the first class. This tree is an illustrative construction showing the shape that non-monotonic behavior allows, not a specific model from the source.

The source illustrates regression with a dataset of historical computer memory (RAM) prices. Each point records a year on the horizontal axis and the price of one megabyte of RAM in that year on the vertical axis. The target, price, is a number that changes over time, which makes it a natural fit for a regression tree.

The price axis in the source is drawn on a logarithmic scale. A logarithmic scale spaces values by their ratio rather than their difference, so equal vertical distances represent equal multiplicative changes. A price that doubles and a price that doubles again look equally far apart on a log axis, even if the absolute price changes are very different. This choice changes how the data looks, and the source uses it on purpose.

Because a log axis converts steady ratio changes into steady visual distances, a curved pattern on a linear axis can look close to a straight line on a log axis. That makes the relationship easier to see. However, the change in appearance is only a change in how the data is displayed. It does not change the underlying values, and it does not give the tree any new ability to continue the pattern past its training data.

Extrapolation means making predictions outside the range of the data the model was trained on. The source states that DecisionTreeRegressor, and all other tree-based regression models, are not able to extrapolate. They make predictions only within the range of the training data. The source treats this as the single most important caveat about regression trees.

The consequence is that a regression tree cannot follow a trend past the edge of its training data. Even when the RAM price data looks linear on a log scale, a tree trained on that history will not simply continue the line into dates it has never seen. Its predictions for those dates are limited by the values it learned from the training range.