Model Evaluation using R^2 Score

A regression model learns from the columns you give it, and those columns are not always the raw values that were recorded. Feature engineering is the process of selecting and transforming raw data into features that are suitable for modeling. It can significantly affect how well a model performs. The same underlying dataset can produce a weak model or a useful one, depending on how its information is presented to the learner.

This article follows one practical example built on a citibike dataset. The example takes a time-based index from that dataset, turns part of it into a feature, fits a regressor, and then reports how well the model performs on held-out test data. The goal is to work out what the reported number means and where it came from. The number itself is a single R^{^2} score, and the feature that produces it is one column called X_hour.

Before reading the explanation, make a guess. Suppose an evaluation reports a test-set R^{^2} score of 0.60 for a model. Decide, in your own words, what that number is telling you about the target and about the model's predictions. Then compare your guess with the reading below.

The raw data in this example is a time-based index on the citibike dataset. The regressor never receives that index directly in the code shown. Instead, the hour component is extracted and reshaped into X_hour, which is the feature matrix the evaluation uses. Everything the score reflects comes from that single feature.

Read the diagram as a chain of hand-offs. Each arrow is a point where the data can change shape or meaning. The extraction step decides which part of the timestamp becomes the feature, so a different choice at that step produces a different model input. The reshape step does not add information; it only changes how the values are arranged so the rest of the pipeline receives them in the layout it expects. The fit and predict steps then turn that feature matrix into a score.

A single timestamp is a point in time, and a regressor cannot split on it the way it can split on a number. Extracting the hour gives a plain integer for each row, such as 8 for a row recorded at 08:15 or 17 for a row recorded at 17:45. Each timestamp in the index becomes one numeric value in the feature column.

This snippet is an illustration of the extraction idea, not the exact code behind the citibike score. The source shows the call and its output but not the full definition of eval_on_features or the code that created the regressor and the target. Treat those pieces as given and focus on the single feature that the first line produces.

The source gives a concrete reading of the value in this example. An R^{^2} score of 0.60 indicates that 60 percent of the variance in the target variable is explained by the model. The score is a proportion between 0 and 1 that is commonly reported as a percentage, so $R^{^2} = 0.60$ means that roughly 60 percent of the variability in the target is accounted for by the model's inputs.

The 60 percent figure refers to held-out data. The model is fit on one set of rows and scored on another set that it did not see during fitting. A test-set score therefore reports how much of the target's variation the model accounts for on rows that were not used to learn it. This is why the evaluation is computed on the test set rather than the training set: the test rows show whether the learned pattern carries over to new data.

When a score is surprising, trace the pipeline step by step from raw index to evaluation and change one thing at a time. Because the score depends on each stage, a change anywhere upstream can alter the score. The worked example below follows one such change.

Not every step carries the same kind of change. The extraction step can change which information reaches the model. The reshape step does not add information; it changes only the layout. When you trace a result, ask of each step whether it alters what the model can see or only how that information is arranged.