Regression Modeling

A regression model learns from the columns you give it, and those columns are not always the raw values that were recorded. Feature engineering is the process of selecting and transforming raw data into features that are suitable for modeling. This matters because feature engineering can significantly affect how well a model performs. The same underlying dataset can produce a weak model or a useful one, depending on how its information is presented to the learner.

This article follows one practical example built on a citibike dataset. The example takes a time-based index from that dataset, turns part of it into a feature, fits a regressor, and then reports how well the model performs on held-out test data. Before reading the explanation, run the example and look at the reported score. The goal is to work out what the number is telling you and where it came from.

Whatever you guessed, the reported value is the one to interpret. The rest of this article explains what the R^{^2} score measures, so that a value such as 0.60 can be read correctly rather than memorized.

The raw data in this example is a time-based index on the citibike dataset. A datetime index holds a full timestamp for each row, but a regressor cannot use a timestamp as a single number in a meaningful way. In the example, the hour component of the index is extracted and reshaped into X_hour. That reshaped result is the feature matrix the evaluation uses. The regressor never receives the index itself.

Read the diagram as a chain of decisions. The extraction step decides which part of the timestamp becomes the feature, so the choice made there determines what information is even available to the model. The reshape step does not add information. It only changes how the values are arranged so the rest of the pipeline receives them in the layout it expects.

The score in this example reflects only the single feature X_hour that the first line produces. Everything the model can explain has to come through that one column. If a pattern in the target depends on information that the feature does not carry, then no regressor fitted on that feature can capture it, no matter how the fitting is done.

This is why the same dataset can yield a weak or a useful model. Imagine the raw index handed to the model in a form it cannot read well, versus the same information turned into a feature that expresses the part of time that matters. The data is identical in both cases. Only the representation differs, and the test-set score is where that difference shows up.

The score is a proportion between 0 and 1, and it is commonly reported as a percentage. So $R^{^2} = 0.60$ means that roughly 60 percent of the variability in the target is accounted for by the model's inputs. The remaining part of the variance is left unexplained by this model.

The 40 percent in the unexplained column is derived directly from the 60 percent figure, since the two shares make up the whole. The score does not say why the remaining variance is unexplained. It could reflect missing information in the feature, or other factors the model does not have access to. The score only reports the share.

When a score is surprising, the source recommends tracing the pipeline step by step from raw index to evaluation and changing one thing at a time. The pipeline in this example has four stages. Each arrow is a point where the data can change shape or meaning, so each arrow is a place where the final score can depend on what happened.

Notice where the result depends on each step. The extraction step sets the information content of the feature. The reshape step leaves that content unchanged but determines the layout the fitting step expects. The fitting step learns a relationship from X_hour, and the evaluation step measures how much of the target that relationship accounts for on held-out data. A problem at any upstream stage shows up later as a different score, but the score alone does not tell you which stage caused it.