Evaluating Regression Models with R Squared

A negative R squared means the model performs worse than simply predicting the average.

Programming

Why a Too-Good Score Is a Warning

Cross-validation is supposed to give an honest estimate of how a model will behave on data it has never seen. That honesty depends on one assumption: the held-out data must stay held out. The moment information from the held-out portion flows back into the way the model is built, the estimate stops being honest. The result is often a score that looks excellent, and that excellence is the problem.

This article follows one concrete way the assumption breaks. A preprocessing step, feature selection, is run on the full dataset before cross-validation begins. The example uses a dataset of 100 samples with 10,000 features, where the features are pure random noise and the target has no real relationship to any of them. Because there is no real signal, any score meaningfully above zero should make you suspicious of the workflow.

What do you think happens?

The features are pure random noise and the target is unrelated to them. What would a leaky workflow's cross-validated R squared most likely look like?

Clearly negative
Close to zero
Suspiciously high and positive

Reveal answer

Answer:

Suspiciously high and positive

The selector was fitted on every row, using the target values, before the folds were created. The columns it kept were chosen with the held-out rows already in view, so the score looks strong even though no real relationship exists.

Where Held-Out Information Enters

Feature selection is often treated as bookkeeping: choose the most useful columns, then evaluate the model on those columns. Many people believe that doing this before cross-validation is safe, because selection looks like ordinary data preparation rather than model fitting. The source's point is that selection uses the target values. A selector that scores columns against the target is already using the outcome, so it is a form of learning from the data.

In the leaky version, the selector is fitted once, on every row, using the target values. Only after that does cross-validation split the already-reduced data into folds, and each fold is scored with a Ridge model. The critical detail is the order of operations. By the time the folds exist, the choice of which features to keep has already been made using information from all of them, including the rows that will later be used as the test fold.

Which step lets information from the test folds reach the feature selector, and at what point in the pipeline does that happen?

Read the diagram as a path. The target values of the test rows travel into the selector before the split happens, and the selector's output is what the model is later evaluated on. The model never trained on the test rows directly, but its inputs were chosen with them in view. That is why the score it earns is not a fair measure of performance on new data.

The Order of Operations Per Fold

The leak is a question of order. In the leaky workflow, selection runs once on the full dataset and the loop only handles model fitting and scoring. In a clean workflow, the selection step is repeated inside every fold, and in each fold it is fitted only on that fold's training rows. The held-out rows are then transformed using what was learned from the training rows, never the other way around.

For each fold, what runs first, what runs second, and which steps are repeated inside the loop versus run once on the full dataset?

Because selection happens inside the loop in the clean version, each fold chooses its own features from its own training rows. The source's random-data example shows the consequence: features that look correlated with the target on the training rows are not correlated with the target on the test rows, so the honest score falls below zero.

Which Rows Each Step Sees

To reason about a leak, it helps to name the rows. Imagine a small version of the problem with 10 rows split into 5 folds of 2 rows each. In each iteration, one fold is the test fold and the remaining four folds form the training set. The diagram below labels each row by its fold, so you can check which rows each selection step is allowed to see.

Which rows are in the training fold and which are in the test fold for each iteration, and when does the selector see each set?

Worked example

Tracing what the selector sees in fold 1

Using the 10-row, 5-fold layout, determine which rows are visible to the selector when fold 1 is the test fold, under the leaky and the clean workflow.

1
Identify the test fold
Fold 1 holds rows 1 and 2. These are the rows the model should be scored on.
2
Leaky workflow selection
The selector is fitted once on all 10 rows, including rows 1 and 2 and their target values. Fold 1's test rows have already influenced which features are kept.
3
Clean workflow selection
The selector is refitted for fold 1 using only the training rows, which are rows 3 through 10. Rows 1 and 2 are not visible to it.
4
Scoring
In both workflows, the Ridge model is scored on rows 1 and 2. Only the clean workflow ensures those rows never shaped the feature choice.

Result

The leaky selector sees all 10 rows, so the fold 1 test rows influence the features. The clean selector sees only the 8 training rows for fold 1.

Rebuilding the Workflow as a Pipeline

A pipeline bundles the feature selector and the model into a single estimator. When that combined estimator is passed to cross-validation, the selection step is no longer a one-time event on the full data. It is repeated inside every fold and fitted only on the training portion. The code below is an illustrative example of the leaky and clean versions on data shaped like the source's: 100 samples, 10,000 random features, and 500 kept columns.

python

import numpy as np
from sklearn.feature_selection import SelectKBest, f_regression
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import KFold, cross_val_score

rng = np.random.default_rng(0)
X = rng.normal(size=(100, 10000))
y = rng.normal(size=100)
cv = KFold(n_splits=5, shuffle=True, random_state=0)

# Leaky: selection sees every row, including each test fold
X_leaky = SelectKBest(f_regression, k=500).fit_transform(X, y)
leaky_scores = cross_val_score(Ridge(), X_leaky, y, cv=cv, scoring="r2")

# Clean: selection is refitted inside each training fold
clean_pipe = make_pipeline(SelectKBest(f_regression, k=500), Ridge())
clean_scores = cross_val_score(clean_pipe, X, y, cv=cv, scoring="r2")

print(leaky_scores.mean(), clean_scores.mean())

import numpy as np
from sklearn.feature_selection import SelectKBest, f_regression
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import KFold, cross_val_score

rng = np.random.default_rng(0)
X = rng.normal(size=(100, 10000))
y = rng.normal(size=100)
cv = KFold(n_splits=5, shuffle=True, random_state=0)

# Leaky: selection sees every row, including each test fold
X_leaky = SelectKBest(f_regression, k=500).fit_transform(X, y)
leaky_scores = cross_val_score(Ridge(), X_leaky, y, cv=cv, scoring="r2")

# Clean: selection is refitted inside each training fold
clean_pipe = make_pipeline(SelectKBest(f_regression, k=500), Ridge())
clean_scores = cross_val_score(clean_pipe, X, y, cv=cv, scoring="r2")

print(leaky_scores.mean(), clean_scores.mean())

Notes on the code

X and y are random noise, so no real relationship exists
The leaky line runs the selector on all 100 rows before any split
The pipeline puts selection and model into one estimator
Passing the pipeline to cross_val_score refits the selector in every fold

Before

X_reduced = SelectKBest(f_regression, k=500).fit_transform(X, y)
cross_val_score(Ridge(), X_reduced, y, cv=cv)

Selector fitted once on every row, then folds created

After

pipe = make_pipeline(SelectKBest(f_regression, k=500), Ridge())
cross_val_score(pipe, X, y, cv=cv)

Selector refitted inside each fold on training rows only

How does the structure of the leaky workflow differ from the pipeline version, and which data does the selector receive in each?

The pipeline does not make the model better. It makes the measurement honest. Keeping 500 out of 10,000 columns is not inherently wrong; selection is a real modeling decision. What makes the leaky workflow fail is that the decision was made with the held-out rows already included in the evidence.

Reading a Negative or Inflated Score

Once you know the mechanism, the score itself becomes evidence. A suspiciously high score on random data is a warning that the workflow is leaking. When you fix the workflow and the score drops to a negative R squared, that is also a signal, and it is about the process rather than a verdict on the model. A negative R squared means the model performs worse than simply predicting the average. On data with no real signal, that is what an honest estimate should reveal.

How do the scores from a leaky workflow and a clean workflow compare, and what does each pattern suggest about the workflow rather than the model?

Mistakes Beginners Make

Common mistake
Treating feature selection as harmless bookkeeping
What it looks like
Calling SelectKBest on the full dataset, then passing the result to cross_val_score.
Why it fails
The selector uses the target values of every row, including the rows that become test folds.
Do this instead
Put the selector and the model into a pipeline and pass the pipeline to cross-validation.
Common mistake
Assuming a high cross-validated score proves the model is good
What it looks like
Reporting a strong R squared on data that is pure random noise and not questioning it.
Why it fails
A suspiciously high score on random data is a diagnostic signal that information may have leaked.
Do this instead
Check the workflow order before reporting the score, and test on data where no signal should exist.
Common mistake
Abandoning a model because its clean score is negative
What it looks like
Dropping the model after the pipeline version returns a negative R squared, without examining the setup.
Why it fails
The negative score is the honest estimate of performance on this data, and it reflects the workflow being measured correctly.
Common mistake
Refitting the scaler or selector on the full dataset before splitting, even when the model is inside a pipeline

Practice: Diagnose the Workflow

MEDIUM

A colleague builds a Ridge model on 100 rows and 10,000 features. They run a feature selector on all rows, keep 500 columns, and then call cross-validation. They report an R squared that looks strong, but the target was generated independently of the features. Identify the step where test-fold information enters, explain why the score is inflated, and describe the change that would make the selector see only training folds.

Hints

Ask which step uses the target values, and when it runs relative to the split.
Think about what a pipeline changes: it moves the selector inside the loop.
The fix is structural, not a change to the Ridge model's settings.

Your attempt (not saved)

Key Takeaways

Information leakage happens when information from test folds shapes the training process, which inflates the performance estimate.
Feature selection run on the full dataset before cross-validation uses the target values of every row, so held-out information reaches the selector.
A pipeline keeps feature selection inside the loop, so each fold selects features using only its training data.
A suspiciously high score on random data is a diagnostic signal about the workflow; a negative score after fixing it is an honest estimate, not a verdict on the model alone.

Related concepts

More concepts starting with E