Evaluating Machine Learning Models on Test Data

Test data is never used to decide which features to keep.

Programming

Why Let an Algorithm Choose the Columns

When you build a machine learning model, you must decide which columns of your data to feed into the algorithm. Sometimes the right choice is obvious. Often it is not, and you are unsure which inputs carry useful signal. Feature selection answers a practical question: can an automatic method pick the useful columns for you? Recursive Feature Elimination, usually called RFE, is one such method. It works by recursively eliminating features until a specified number is reached.

In this article you will first look at the behavior of a short RFE program, then work out from that behavior how RFE makes its choices. Keep one question in mind as you read: if RFE keeps retraining a model while it removes features, what does that tell you about its cost? The answer comes back at the end of the article when we compare RFE with other selection approaches.

Watching the Elimination Loop

RFE is wrapped around an estimator, which is the model that RFE trains again and again while it decides what to remove. The loop is simple to state. Fit the estimator on the current set of features. Use the fitted model to decide which feature to drop. Remove it. Repeat until only the number you asked for is left. The argument n_features_to_select tells RFE where to stop.

Starting with six features and a target of three, how many features remain after each round, and when does the loop stop?

The diagram shows the shape of the process. Each pass through the loop removes one feature and leaves a smaller feature set for the next pass. Notice that the loop does not stop after a single decision. It keeps going until the target count is reached, so the number of rounds is set by how many features you drop, not by how good any one feature looks.

Worked example

Tracing RFE from six features down to three

A dataset has six feature columns. You ask RFE to select three. Trace what happens.

1
Start
RFE begins with all six features. The estimator is trained on this full set.
2
First elimination
The fitted estimator is used to decide which feature to remove. One feature is dropped, leaving five. The estimator is trained again on the five remaining features.
3
Second elimination
Another feature is dropped, leaving four. The estimator is trained again on those four.
4
Third elimination
A third feature is dropped, leaving three. Three drops means three rounds of retraining, which is exactly the cost the source describes.
5
Stop
The target of three has been reached, so the loop ends and the three surviving features form the selected set.

Result

Three features are selected. The estimator was retrained once for each feature dropped.

Reading the Selected-Feature Mask

After fit has run, the selector stores its result. The method get_support returns a mask with one entry per original feature. Each entry is True if that feature was kept and False if it was eliminated. The mask is aligned by position, so position 0 in the mask describes column 0 of your original data, position 1 describes column 1, and so on. Mapping the mask back to column names is how you learn which features survived.

python

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import RFE

X, y = make_classification(n_samples=500, n_features=20, n_informative=5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

estimator = RandomForestClassifier(n_estimators=100, random_state=42)
selector = RFE(estimator, n_features_to_select=5)
selector.fit(X_train, y_train)

mask = selector.get_support()
print(mask)
print(mask.sum())

feature_names = [f'feature_{i}' for i in range(20)]
selected = [name for name, keep in zip(feature_names, mask) if keep]
print(selected)

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import RFE

X, y = make_classification(n_samples=500, n_features=20, n_informative=5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

estimator = RandomForestClassifier(n_estimators=100, random_state=42)
selector = RFE(estimator, n_features_to_select=5)
selector.fit(X_train, y_train)

mask = selector.get_support()
print(mask)
print(mask.sum())

feature_names = [f'feature_{i}' for i in range(20)]
selected = [name for name, keep in zip(feature_names, mask) if keep]
print(selected)

Output (expected)

A 20-item array of True and False values. The sum of the mask is 5, and the selected list contains five names such as feature_0 or feature_3, depending on the data.

Notes on the code

RandomForestClassifier with 100 trees and random_state 42 is the estimator RFE retrains while it removes features.
n_features_to_select=5 tells RFE where to stop.
get_support returns one True or False entry per original column.
zip pairs each column name with its mask entry so the surviving names can be read directly.

The code above is an illustrative program written for this article. It uses a synthetic dataset, so the exact names that survive depend on the data, but the structure of the result is fixed. The mask will contain exactly as many True entries as you requested, and each True position corresponds to one surviving column.

Each position in the mask lines up with one original feature column. Which column names survive when the mask is True at positions 0, 2, and 3?

In this illustrative five-column case, the mask reads True, False, True, True, False. Reading it position by position gives age, tenure, and clicks as the survivors, while income and region were eliminated. The mask is only a record of the decision. It does not explain why each feature was kept, so the reasoning lives in the elimination loop you traced earlier.

Fitting on Training Data and Transforming Test Data

Once RFE has fit, the selector can shrink your data to the chosen features. The transform method reduces a dataset to the selected columns, in the correct order. The important rule is that the selector is fitted only on training data. Test data is never used to decide which features to keep. Instead, test data is passed through transform so it has the same columns the model was trained on.

Which data fits the selector, which data is only transformed, and where does the test set enter the pipeline?

python

from sklearn.linear_model import LogisticRegression

X_train_small = selector.transform(X_train)
X_test_small = selector.transform(X_test)

model = LogisticRegression(max_iter=1000)
model.fit(X_train_small, y_train)
print(model.score(X_test_small, y_test))

print(selector.score(X_test, y_test))

from sklearn.linear_model import LogisticRegression

X_train_small = selector.transform(X_train)
X_test_small = selector.transform(X_test)

model = LogisticRegression(max_iter=1000)
model.fit(X_train_small, y_train)
print(model.score(X_test_small, y_test))

print(selector.score(X_test, y_test))

Output (expected)

Two test accuracy values, one from the logistic regression on the reduced features and one from the random forest inside RFE on the selected features.

Notes on the code

transform is applied to the training data and the test data, but fit is never called on the test data.
LogisticRegression must be imported from sklearn.linear_model before use.
model.score on the reduced test set reports test accuracy for the logistic regression.
selector.score scores the random forest inside RFE using only the selected features.

The source follows this same logic. It transforms the training data, fits a logistic regression on the reduced training set, and scores that model on the reduced test set. It also scores the random forest inside RFE directly, which uses only the selected feature set. In the source's example, the random forest and a logistic regression on the selected features scored the same, which shows that a linear model can match the forest once the right features are chosen.

What RFE Costs Compared with Other Approaches

RFE is usually a better-performing choice than simpler selection methods in the source's experiment, but its power comes at a price. Because a random forest is trained once for each feature that is dropped, the total work grows with the number of eliminations. The source describes RFE as computationally expensive for this reason, and notes that it took significantly longer to run than the model-based selection it was compared with.

RFE trains a model for each dropped feature, while model-based selection trains one model. How do the three approaches differ in what they train and what they achieve?

Approach	Training effort	Result reported in the source
RFE with a random forest	Retrains the estimator for each dropped feature	Selected features that performed better than the other two approaches, but missed one feature
Model-based selection	Trains one model	Faster, but its selected features scored lower than RFE's
Univariate selection	Not retrained in the same loop	Selected features that scored lower than RFE's

Trade-offs described in the source's comparison

The source also notes that RFE does not guarantee a perfect selection. In its experiment, one feature was still missed by RFE, so even its best result is not guaranteed to be ideal. RFE is therefore worth its cost when the performance gain matters, your dataset is manageable to retrain repeatedly, and you are genuinely unsure which inputs to keep. When speed matters more, or when a quick first pass is enough, a single-pass or model-based method may be the better trade.

Mistakes When Applying RFE

Common mistake
Fitting RFE on the full dataset before splitting
What it looks like
selector.fit(X, y) is called on all rows, and only then is the data split into train and test.
Why it fails
The test set has influenced which features were kept, so the test score is no longer an honest estimate for unseen data.
Do this instead
Split first, then call selector.fit(X_train, y_train) using only training rows.
Common mistake
Calling fit on the test data
What it looks like
selector.fit(X_test, y_test) is run to get the test set in the right shape.
Why it fails
Fitting learns from the data it receives. Test data should only be transformed, never used to make decisions.
Do this instead
Call selector.transform(X_test) after fitting on training data.
Common mistake
Reading the mask as column names
What it looks like
```
print(selector.get_support()) is treated as a list of kept feature names.
```
Why it fails
The mask holds True or False values aligned by position, not the names of the columns.
Do this instead
Pair the mask with the original column names, for example with zip, to find which features survived.
Common mistake
Forgetting that the mask is aligned with the original columns
What it looks like
The mask is applied to a dataset whose columns were reordered after fitting.
Why it fails
Each position in the mask refers to a specific original column, so reordering breaks the mapping.
Do this instead
Keep the column order identical between fitting and reading the mask.
Common mistake
Choosing RFE without considering its cost
What it looks like
RFE is run on a very large dataset with a slow estimator for a quick first experiment.
Why it fails
RFE retrains its estimator for each dropped feature, so runtime grows with the number of eliminations.
Do this instead
Estimate the runtime first, and compare against a faster approach if speed matters more than the expected gain.

Practice: Trace the Selection

MEDIUM

Suppose RFE is configured with n_features_to_select=3 on a dataset with eight columns, and the mask returned by get_support has True at positions 1, 4, and 6. Before reading the answer, predict: how many times was the estimator retrained, and which column names survive if the columns are named col0 through col7? Explain your reasoning using the elimination loop.

Hints

Count how many features must be dropped to go from eight to three.
Each dropped feature means one retraining of the estimator.
Mask positions are zero-indexed, so position 1 is col1.

Your attempt (not saved)

MEDIUM

Write the three lines of code that fit an RFE selector on training data only, transform both the training and test sets, and print the test score of a logistic regression trained on the reduced training set.