Feature Engineering for Time-Based Data
When a feature value lands inside the range of training values, the tree has a region to send it to.
The Forgiving-Model Assumption
When you start a regression problem, a random forest is a tempting first choice. Random forests require very little preprocessing of the data, so you can often hand over raw feature values and get a model that makes sensible predictions without much tuning. That convenience is real, but it rests on a hidden assumption: that a model which is easy to set up will also be reliable on any new data you feed it.
This article tests that assumption. The specific misconception to watch for is that a random forest can extrapolate to feature ranges it never saw during training. The source material shows exactly why that belief fails, and it also shows how a second model can serve as a comparison point, so that you judge the first model's behavior instead of trusting it.
What do you think happens?
Suppose you train a random forest regressor on a time-based feature whose training timestamps all come from earlier in a dataset. You then ask it to predict targets for a test set whose timestamps all fall after the last training timestamp. If the model had learned a smooth trend over time, what should the predictions look like?
Reveal answer
A smooth trend would be expected only if the model could extend what it learned. A random forest cannot do that for feature values beyond its training range, so its test predictions flatten out instead.
The source states that trees and random forests cannot extrapolate. For feature values beyond the training range, the model predicts the target of the closest training point, which is why a flat line appears where a trend might have been expected.
How a Tree Routes an Input
A random forest is built from many decision trees. Each tree makes a prediction by routing an input to a region that was populated during training. The forest then combines the trees into one prediction. The important detail for this article is what happens at the routing step: an input has to land in a region that training filled in order to receive a learned value.
When a feature value lands inside the range of training values, the tree has a region to send it to. When the value falls outside that range, no trained region covers it, and the tree cannot invent a value for it. The tree does not reason about the trend that the feature seems to follow. It only knows the regions that training produced.
Trace the diagram from the top. The first question is whether the test value sits inside the training range. If it does, the input follows the splits to a populated leaf and returns a learned value. If it does not, the path leads to the closest training boundary, and the output is inherited from that boundary. The key change happens at the moment the test value leaves the training range.
Reading the Forest's Two Results
The source describes two results that look contradictory at first. The predictions on the training set are quite good, which is typical for random forests. The predictions on the test set, however, form a constant line, and the R2 on the test set is -0.03. A model can fit its own training data closely while failing completely on unseen data.
The lesson here is about where you look. The failure was not caused by poor training. The source states the cause directly: the test timestamps lie outside the range of the training timestamps. The combination of the feature and the model produced the failure, and the training-set results could not reveal it.
Locating the Divergence in the Forest Comparison
A random forest regressor is evaluated on a time-based feature and gives a flat test-set prediction with an R2 of -0.03. Where should you look first to explain the failure?
- 1Identify the feature
The forest in the source uses POSIX time as its single feature, so the input is a timestamp rather than a categorical or cyclical field.
- 2Compare test values to the training range
Check whether the test timestamps fall inside the span covered by the training timestamps. In the source, they all fall after the last training timestamp.
- 3Locate the divergence
The divergence sits in the test-set predictions, not the training-set predictions. Training values were inside the range, so they routed to learned regions; test values were outside it.
- 4Judge the model last
Only after confirming the range mismatch should you conclude the model learned little useful signal for the unseen period. The R2 of -0.03 is the evidence that it did not generalize.
The failure is located in the test-set feature values that lie beyond the training range. The forest did not learn a trend that it could extend, so its test predictions are a constant line.
Interpreting an R2 Near Zero
R2 is a score that compares a model's errors with a simple baseline. The source reads a test-set value of -0.03 as meaning the model learned nothing useful. A value near zero means the model's predictions are about as informative as the baseline; a value below zero means the model is doing worse than that baseline on the data being scored. The source does not give a formula here, so the interpretation to carry forward is the comparison itself: the question is whether the model beats a flat guess on unseen data.
Comparing Models Without Confusing the Causes
The source then changes two things at once: the model and the features. It uses a LinearRegression model, imported from the linear model module, and passes it to the same evaluation helper, eval_on_features, with different input features: day of week and hour of day. The source presents linear regression as a comparison to the random forest regressor, not as a claim that one model is always better.
Because the features changed along with the model, you cannot attribute the difference in behavior to the model type alone. The forest struggled when its only feature was a time value that ran past the training data. The linear model works on a different feature set, so its result gives a reference for judging whether the trouble lay with the forest, the feature, or both. Read the linear regression output directly rather than assuming its quality from the forest's result.
| Aspect | Random forest regressor (first call) | Linear regression (second call) |
|---|---|---|
| Feature | POSIX time, a single timestamp feature | Day of week and hour of day |
| Configuration noted in source | n_estimators set to 100, random_state set to 0 | Imported from the linear model module |
| Test-set behavior reported | Constant line, R2 of -0.03 | Read the caption and output directly |
| Role in the comparison | The model under diagnosis | A reference for judging the forest and the feature |
The two evaluation calls differ in both model and features, so the comparison isolates neither cause alone.
The fixed random state in the forest call matters for the comparison. It makes the forest reproducible, so the same run gives the same result each time. That reproducibility lets you trust that a difference you observe reflects the model and features, not randomness in the forest itself.
Tracing a Divergence Step by Step
The following example is illustrative and uses invented numbers to show the tracing method. Suppose a forest is trained on hourly values for hours 0 through 23 of a day and then asked about hours 24 through 30. The hours 24 through 30 are just past the training maximum, so they are outside the range the forest saw.
Trace the forest's routing for each test hour, as in the flowchart. Hour 12 lies inside the training range, so it routes through the splits to a populated leaf and returns a learned value. Hour 24 lies just outside the range, so no trained region covers it and it inherits the output of the boundary at hour 23. Hours 25 through 30 do the same, so the predictions for them repeat the boundary value and form a flat stretch.- 1Hour 12 is inside the training range and reaches a populated leaf.
- 2Hour 24 is outside the training range and inherits the nearest boundary output.
- 3Hours 25 through 30 inherit the same boundary output, producing a flat line.
- The tracing method checks the range first, then the route, then the output.
Notice what this trace tells you before any score is computed. You can predict where the divergence will appear: at the first test value outside the training range. The method is the same one the source recommends: trace the test values against the training range first, and only then judge the model's predictions.
Mistakes Beginners Make With Time Features
Common mistake Assuming a random forest will continue a trend past the training data
What it looks likeTraining on earlier timestamps and expecting a rising prediction for later timestamps.
Why it failsTrees route inputs to regions populated during training and cannot invent values outside that range.
Do this insteadCheck whether the test feature values fall inside the training range before expecting a trend.
Common mistake Judging a model by its training-set predictions
What it looks likeSeeing good training predictions and concluding the model will work on new data.
Why it failsA model can fit its own training data closely while failing on unseen data, as the source's forest did.
Do this insteadAlways read the test-set results directly before judging the model.
Common mistake Reading a negative test R2 as a small error
What it looks likeSeeing an R2 of -0.03 and treating it as close to a good score.
Why it failsA value near zero or below zero means the model has learned little useful signal on that data.
Do this insteadInterpret the value against the baseline and conclude the model did not generalize.
Common mistake Attributing a comparison difference to the model alone
What it looks likeCrediting the linear model's better behavior entirely to its algorithm.
Why it failsThe source changed both the model and the features, so the two effects are tangled together.
Do this insteadSeparate the effect of the model from the effect of the features before drawing a conclusion.
Practice Tracing a Test Set
A forest is trained on feature values from 10 to 50. You evaluate it on a test set with feature values 45, 52, and 60. Which test values can the forest route to learned regions, which ones inherit a boundary output, and where would you expect the predictions to flatten? Then state what a test-set R2 near zero would tell you about the model.
Hints
- Compare each test value to the training range of 10 to 50 first.
- A value inside the range can reach a populated leaf; a value outside it cannot.
- Flattening begins at the first value beyond the training maximum.
Key Takeaways for Time-Based Features
A random forest needs little preprocessing, but that convenience does not mean it handles every input range. Trace the test values against the training range before trusting the model's predictions.
At a glance
- A random forest can look accurate on training data and still fail on a test set when the test feature values fall outside the training range.
- Trees and random forests cannot extrapolate: for values beyond the training range, the model predicts the target of the closest training point, producing a flat line.
- A test-set R2 near zero or below zero is evidence that the model learned little useful signal on that data.
- When comparing models, separate the effect of the model from the effect of the features, and read the test-set results directly.
- The first diagnostic step is to trace the test values against the training range, then judge the predictions.