Interpreting R-Squared in Model Evaluation

A common first move when starting a regression problem is to reach for a random forest. The reasoning is easy to follow: random forests require very little preprocessing, so you can often hand over raw feature values and get a model that makes sensible predictions without much tuning. That convenience is real, but it rests on a hidden assumption, namely that a model which is easy to set up will also be reliable on new data. This article tests that assumption and shows how to read the evidence when it fails.

The rest of this article follows that surprise to its cause. You will trace how a single input moves through a tree, see why inputs beyond the training range inherit a boundary value, learn how to read a train-versus-test gap through R2, and practice diagnosing a comparison by locating the exact point where a prediction departs from what you expected.

A random forest is built from many decision trees. Each tree makes its prediction by routing an input down a set of splits until it lands in a region that was populated during training. The forest's prediction is built from those individual tree predictions, so understanding one tree's routing is the first step to understanding the whole forest.

The routing depends entirely on whether the input falls inside the range of values the training data covered. When a feature value lands inside that range, the tree has a region to send it to. When the value falls outside the range, no trained region covers it. The tree cannot invent a value for an input it has no region for, so it has nowhere principled to go.

Read the diagram as a question you ask at every split. While the answer keeps coming back yes, the prediction is grounded in training examples. The first time the input leaves the training range, the path no longer corresponds to any region the model learned, and the output is taken from the edge of the training data instead.

Extrapolation means predicting beyond the range of inputs the model has seen. Trees and random forests do not extrapolate. For a feature value beyond the training range, the model predicts the target of the closest training point. Across many such inputs, the predictions stop following any trend and settle at a boundary value.

The source describes two results that look contradictory at first. The predictions on the training set are quite good, which is typical for random forests. The predictions on the test set, however, form a constant line, and the R2 on the test set is -0.03. R2 measures how much of the variation in the target the predictions explain, so a value near zero means the model explains almost none of that variation on the data it was scored on.

The source reads the test-set R2 of -0.03 as meaning the model learned nothing useful for that test set. A value near zero indicates that the predictions do no better than the simplest baseline of always predicting the average target. A value below zero indicates the predictions are worse than that baseline. In both cases the signal the model captured does not carry over to the new data.

The lesson in this case is about where you look. The failure was not caused by poor training. It was caused by the combination of the feature and the model: the test timestamps lie outside the range of the training timestamps. A gap between training and test results tells you that something about the data's coverage or the feature choice needs investigating.

The source builds its comparison from two calls to the same evaluation helper, eval_on_features, which receives a feature matrix, a target, and a model. In the first call, the feature is the POSIX time and the model is a RandomForestRegressor with n_estimators set to 100 and random_state set to 0. The fixed random state makes the forest reproducible, so the same run gives the same result each time.

The source then changes two things at once: the model and the features. It uses a LinearRegression model, imported from the linear model module, and passes it to the same helper with different input features, namely day of week and hour of day. Linear regression serves here as a comparison point, which is the purpose of the comparison rather than a claim that one model is always better.

Because the features changed along with the model, you cannot attribute the difference in behavior to the model type alone. The random forest struggled when its only feature was a time value that ran past the training data. The linear model's result gives a reference for judging whether the trouble lay with the forest, the feature, or both. Its caption describes predictions made from day of week and hour of day, and you should read that output directly rather than assume its quality from the forest's result.