Feature Representation and Model Choice

Imagine you have a single continuous input feature and a continuous output that rises and falls as the feature changes. You want to fit two different regression models to the same training points and then draw both predictions on top of those points. One model is a linear regression, and the other is a decision tree regressor. Before reading any explanation, make a prediction: which of the two curves will follow the training points more closely, and why?

The source frames this as a small experiment. The data is the wave regression dataset from Chapter 2, which has only one input feature. The plot you produce compares the two prediction curves over the training points. Your task is to read the plot and identify where each curve sits relative to the points. Memorizing the output is not the goal. The goal is to understand why the two curves look the way they do.

A linear model is limited by its form. When there is a single input feature, the relationship between input and output is a line. A line has exactly two numbers that you can choose: one slope and one intercept. No choice of those two numbers can make the line curve up and down to chase a wavy pattern. So the linear regression curve in this comparison is a straight line, even when the training points clearly bend.

Trace the state of this model step by step. Start with the training points, which form a curved cloud. The linear model then searches only among straight lines, so its final state is one slope and one intercept. Whatever the points do, the prediction has to stay on that single line across the whole input range.

This limitation belongs to the model family, not to a bad fit. A linear model that looks poor on wavy data is doing exactly what it was built to do. When you see a straight prediction line cutting through a curved cloud of points, the first explanation to consider is the model's form, before you blame the code or the data.

A decision tree regressor does not have a fixed functional form like a line. Instead, it builds its prediction by making a sequence of splits on the input feature. Each split cuts the range of the feature into regions, and the tree assigns an output value to each region. Because the prediction is built region by region, the tree can follow bends and irregular patterns that a single straight line cannot.

Follow the state as the tree is built. At the start there is one region covering the whole feature range. The first split divides that range in two. Each further split divides one of the existing regions again. The final state is a set of intervals along the feature axis, and each interval carries a single output value. Plotted together, those values form a stepped, piecewise-constant curve rather than a smooth line.

Binning, also called discretization, splits one continuous feature into multiple features. The source presents it as one way to make a linear model more powerful on continuous data. You choose a set of edges along the feature's range. Each resulting interval is a bin. Each sample is then represented by indicator features that record which bin its value falls into, with a 1 for the bin it belongs to and a 0 for the others.

The key change is what the linear model receives. A linear model still produces a line in the features it is given. When the single continuous feature is replaced by several derived indicator features, the model is no longer restricted to one slope across the entire input range. Each indicator feature can carry its own weight. The combined prediction can therefore rise and fall in different regions, producing a stepped curve. The model is still linear, but the representation of the data has changed.

Notice what this illustration shows and what it hides. The original value is replaced by indicator features that record which range it falls in. After this change, only the bins are visible to the linear model, not the raw values. The model can no longer see how far a sample sits inside its bin, only which bin it is in.

The source is careful about one qualification. A decision tree's ability to capture complex patterns depends strongly on how the data is represented. The same tree can look powerful or weak depending on the feature representation it receives. So the central question is not only which model is more capable. The question is how each model consumes the feature it is given.

When a model comparison gives a result you did not expect, the source recommends working through the pipeline in order instead of guessing. The check runs in three stages. Start with the feature representation: how many input features are there, and are they continuous as given or transformed? Next, identify the model family and what shape it can express. Only then compare the predicted curves with the training points. The first point where your expectation and your reasoning disagree tells you where to look.