Train and Test Splits for Regression

When you begin a regression problem, a random forest is a natural first choice. Random forests require very little preprocessing of the data, so you can often hand over raw feature values and get a model that makes sensible predictions without much tuning. That convenience is real, but it hides an assumption: that a model which is easy to set up will also be reliable on new data. This article tests that assumption.

The misconception to watch for is that a random forest can extrapolate to feature ranges it never saw during training. The source material shows why that belief fails, and it also shows how a second model can serve as a comparison point, so that you judge the first model's behavior rather than trusting it. The goal of this article is to teach you where to look when a split-based evaluation goes wrong.

A train and test split divides the dataset into two disjoint parts before any fitting happens. The training subset is the part the model is fitted on. The held-out subset is kept aside and used only for evaluation. Every row lands in exactly one of the two groups, so the test rows are unseen by the model during training. Follow the flow in the diagram below to see what happens at each step.

The key point of the partition is that the test rows play no part in fitting. Their only role is to measure how the fitted model behaves on data it has not seen. Keeping that role clean is what makes the test-set result informative.

The source describes two results that look contradictory at first. The predictions on the training set are quite good, which is typical for random forests. The predictions on the test set, however, form a constant line, and the R2 on the test set is -0.03. A model can fit its own training data closely while failing completely on unseen data. The training result tells you the forest memorized its training rows well; it does not tell you that the forest will generalize.

The gap between the two columns does not come from careless training. The source states the cause directly: the failure was caused by the combination of the feature and the model, and the test timestamps lie outside the range of the training timestamps. To find that cause, you have to look at the feature values of the test rows, not only at the score.

A random forest is built from many decision trees, and each tree makes a prediction by routing an input to a region that was populated during training. A row enters at the top of the tree, and each split decision sends it down one branch until it reaches a leaf. The leaf's value is what the tree contributes to the prediction. The forest combines the outputs of its trees to produce a single prediction.

This routing works only when there is a trained region to send the input to. When a feature value lands inside the range of training values, the tree has a region to route through. When the value falls outside that range, there is no trained region that covers it, and the tree cannot invent a value for it. That is the mechanism behind the failure described above.

Trees, and therefore random forests, cannot extrapolate to feature ranges outside the training set. For a feature value beyond the training range, the model predicts the target of the closest training point. The output is simply inherited from the boundary of the training data, which is why a flat line appears where you might have expected a trend to continue. Follow the chain of reasoning in the diagram below; the key change happens at the moment the test value leaves the training range.

The divergence therefore sits at the first decision. Everything below that check is ordinary tree routing. Once the value falls outside the training range, the path leads to the boundary, and the prediction stops responding to the input in the way a trend would require.

The R2 score summarizes how well a model's predictions match the targets, compared with a simple baseline. The baseline is a model that always predicts the mean of the target. A model with an R2 near 0 has done about as well as that baseline, which means it has learned little useful signal. A negative R2 means the model does worse than always predicting the mean. The source's test-set value of -0.03 is in this range, and the source reads it as evidence that the model learned nothing.

The source runs the same evaluation helper, eval_on_features, twice. The helper takes a feature matrix, a target, and a model. In the first call, the feature is the POSIX time and the model is a RandomForestRegressor with n_estimators set to 100 and random_state set to 0. In the second call, the source changes two things at once: it uses a LinearRegression model, imported from the linear model module, and passes different input features, namely day of week and hour of day. Linear regression here is a comparison point, not a claim that one model is always better.

A forest's good training fit and poor test fit can coexist, and the divergence is usually located in the test-set feature values. Trace those values against the training range before judging the model.