Feature Engineering with Domain Knowledge

A random forest regressor is one of the easiest models to reach for. It needs very little preprocessing, so a raw feature can often be fed in without much preparation. That ease of use is appealing, but it does not guarantee that the model can predict feature values it has never seen. This article uses a bicycle rental example to show how a model can look sound on paper and still fail on new data, and why the fix lies in the feature rather than in the forest's settings.

Imagine predicting how many bicycles a rental service will rent out on any given day. You have a history of daily rental counts and you want a model that can predict counts for days it has not seen. The first 184 days go into the training set, and the remaining days form the test set. The model is therefore judged on time periods it never learned from, which is exactly the situation that makes the feature's range important.

The first feature used is POSIX time, a single number that counts the time elapsed since a fixed reference moment. Because it is a plain number, the random forest accepts it directly, and the setup looks reasonable. On the training data the predictions look good, which is typical for random forests. The surprise appears when the test predictions are checked against the real rentals.

Many learners assume that a model trained on a rising trend will keep rising when asked about later dates. POSIX time always increases, so it seems natural that a model fitted on earlier timestamps should extend the pattern forward. This belief is wrong for tree-based models, and the reason is specific to how they handle feature values.

The timestamps in the test set are later than every timestamp in the training set. When the forest receives one of those values, it has no training examples in that region of feature space to draw on. It cannot build a prediction from the patterns it learned. Instead, it reuses the target value of whichever training point sits closest, and in this case that is the last moment of observed data. The prediction therefore stops rising at the boundary of the training range.

The divergence is located at the boundary between training and test feature values. The training predictions are not where the problem lies. Following the fit and evaluation step by step, the point where the test inputs leave the range the model has actually seen is the point where its predictions stop tracking reality.

The regressor's score method reports the coefficient of determination, written $R^{2}$, on whatever data you pass in. In this setup the call is made on the test set, so the value describes how well the fitted model explains the test targets. A positive value means the model captures some of the variation in the target. A negative value means the model did worse than a simple baseline that always predicts the mean of the target. The source summarizes this as the model having learned nothing useful.

A near-zero negative $R^{2}$, like the value in the rental example, is therefore a diagnostic signal rather than a small disappointment. Read it alongside a plot of predictions. A flat predicted line against a varying test curve tells the same story as the number. It also points to the cause: the feature, not the number of trees.

Once the cause is clear, the remedy is to bring in domain knowledge. Looking at the rental figures in the training data, two factors appear to matter a great deal: the time of day and the day of the week. Rentals rise and fall with the hour and with the weekly rhythm, and both of those patterns repeat. A repeating feature has values that recur inside the training range, so the forest can learn from them and apply the learned pattern to test days.

This is the key contrast. A raw timestamp keeps producing new values, while a time-of-day feature cycles through the same values again and again. Hour 9 on a test day is the same hour 9 the model saw during training, so the forest has something it can actually use. Day of the week works the same way: Monday in the test period matches Monday in the training period.

The plan has two parts. First, add the new domain features. Second, drop POSIX time, because it cannot teach the model anything useful in this setting: its values for the test period never occur in the training period. The source begins with only the hour of the day. With that feature alone, the predictions take on the same pattern for each day of the week. This is an improvement over the flat line, because the model now responds to a feature whose values the test set shares.

Notice that the forest itself does not change. The model settings stay the same, and only the representation of time changes. This is the core lesson of the topic: when a model fails to generalize, examine what each feature lets it learn before adjusting the algorithm.