Train and Test Splits for Time-Based Data

Imagine you want to predict how many bicycles a rental service will rent out on any given day. You have a history of daily rental counts, and you want a model that can predict counts for days it has not seen. To judge the model fairly, the first 184 days go into the training set and the remaining days form the test set. The model is then scored on time periods it never learned from.

A random forest regressor looks like an excellent first choice. Random forests need very little preprocessing, so a raw feature can be fed in with almost no preparation. The feature in this setup is POSIX time, a single number that counts the time elapsed since a fixed reference moment. Because the model accepts it directly, the whole setup looks reasonable on paper.

The surprise comes when you check the results. Predictions on the training data look good, which is typical for random forests. Predictions on the test data do not follow the real rentals at all. The fix is not to tune the forest. The fix is to rethink the feature.

Before explaining the forest's behavior, trace what the feature values look like on each side of the split. A chronological split puts the training days first and the test days after them. Nothing is shuffled, so every test timestamp is later than every training timestamp.

Now the mismatch is visible. The POSIX time values for the test period lie above the largest value the model ever saw. The question is what the model does with a value that sits outside its experience. Follow the feature values step by step through the fit and the prediction, and note the point where the test inputs leave the range the model has seen. That boundary is where predictions stop tracking reality.

A random forest makes its prediction by asking each of its trees for an output and then averaging those outputs. Each tree sends an input down to a leaf, and that leaf holds a target value learned from training points. A leaf can only hold values that came from the training data, so the forest cannot output anything beyond the training targets it has actually stored.

Because no leaf holds a value from beyond the training range, every tree that receives a test timestamp above the maximum falls back to values from the end of the training period. Averaging those outputs gives a prediction that stays near the last observed level. This is why the predicted line flattens while the real test curve keeps moving. The forest is not broken. It is doing exactly what its training data allows.

The regressor's score method reports the coefficient of determination, written $R^{2}$, on whatever data you pass in. Here the call is made on the test set, so the value describes how well the fitted model explains the test targets. A positive value means the model captures some of the variation in those targets. A negative value means the model did worse than the simple baseline of always predicting the mean of the test targets. In that case the model has learned nothing useful.

A near-zero negative value is not a small disappointment to shrug off. It is a diagnostic signal. Pair the number with a plot of predictions. If the predicted line is flat while the test curve varies, the plot tells the same story as the number, and it also points to the cause. The clue is in the feature, not in the number of trees. Adding trees or adjusting settings will not fix a feature whose test values fall outside the training range.

Once the cause is clear, the remedy comes from domain knowledge. Looking at the rental figures in the training data, two factors appear to matter a great deal: the time of day and the day of the week. Rentals rise and fall with the hour, and they follow a weekly rhythm. Both patterns repeat. A repeating feature has values that recur inside the training range, so the forest can learn from them and apply the learned pattern to test days.

The plan has two parts. First, add the new features. Second, drop POSIX time, because in this setting it cannot teach the model anything useful. Starting with the hour of the day alone already changes the picture. The predictions now take on the same pattern for each day of the week instead of a flat line, because the model responds to a feature whose values the test set shares.