Concepts / Training and Test Splits in Supervised Learning

Training and Test Splits in Supervised Learning

The same scaling transformation must be applied to the training set and the test set, so that the model sees both sets through one set of rules.

  • Programming

Why One Rule Must Govern Both Sets

When you build a supervised model, you split your data into a training set and a test set, and then you scale both. Scaling transforms data so that it falls into a common range, often between 0 and 1. This matters because features with very large ranges can dominate a model. It is easy to assume that each set can be scaled on its own, using whatever minimum and maximum it happens to contain. That assumption is wrong.

The source is direct about the requirement: it is important to apply exactly the same transformation to the training set and the test set for the supervised model to work on the test set. The summary of that requirement is that the model sees both sets through one set of rules. This article explains what that means, shows what goes wrong when the rule is broken, and gives you a way to check your own code.

Before reading on, make a prediction on your own. Suppose a test set contains the values 2 and 8, and you scale it using its own minimum of 2 and its own maximum of 8, as if it were a separate dataset. What do you expect the value 2 to become? Write down your guess so you can compare it with the correct behavior later in this article.

Following the Learned Minimum and Range

MinMaxScaler works in two stages. The fit step looks at data and records what it needs, which for this scaler is the minimum and the range of each feature. The transform step uses those recorded values to convert data. Fitting is where the scaling parameters are learned, and transforming is where they are applied. The required workflow is to fit the scaler on the training data, and then transform both the training data and the test data with that same scaler.

Training set0, 5, 10fitlearns min and rangeFitted scalerone set of rulestransformapplies the same rulesScaled test set0.2, 0.8Test set2, 8inputrecordsrulesentersconverts
How do the minimum and maximum learned from the training data travel into the step that scales the test data?

Read the diagram as a single direction of information. The training set is the only source of the learned minimum and range. The test set enters at the transform step, where it is converted using those same values. Nothing flows from the test set back into the scaler. The test set is a new batch of data that must be read through the rules the model learned from the training set. It does not get to define new rules for itself.

python
Output (expected)
[0.  0.5 1. ]
[0.2 0.8]
Notes on the code
  • X_train holds the training values 0, 5 and 10
  • X_test holds the test values 2 and 8
  • fit is called only on the training data, so the scaler records a minimum of 0 and a range of 10
  • transform is applied to both sets using that one recorded rule
Worked example

Tracing the correct test values by hand

The scaler was fitted on the training values 0, 5 and 10. Find the scaled value of the test value 2.

  1. 1
    Recorded minimum

    The fitted scaler learned a minimum of 0 from the training set.

  2. 2
    Recorded range

    The fitted scaler learned a range of 10, the training maximum of 10 minus the training minimum of 0.

  3. 3
    Apply the rule

    The test value 2 is converted using the minimum of 0 and the range of 10, the same values used for the training data.

  4. 4
    Read the result

    The test value 2 lands at 0.2, a position inside the 0 to 1 range the model learned.

Result

The test value 2 becomes 0.2, and the test value 8 becomes 0.8.

Reading Scaled Values as Positions

It helps to see the mapping as a lookup. The fitted scaler carries one rule, built from the training minimum and range, and every value that passes through transform is looked up under that single rule. The diagram below lists the raw values from the small example on the left and the scaled value that the single fitted scaler assigns to each one on the right. The training and test values appear on the same scale because they share one rule.

Raw 00.0Raw 20.2Raw 50.5Raw 80.8Raw 101.0
Where does each raw value land on the 0 to 1 scale when the training rule is used for every value?

Notice that the test values 2 and 8 sit between the training values, and their scaled positions are consistent with the training values around them. That consistency is what consistent scaling protects. The relative positions of the data points are the thing that matters. The source's own visual makes the same point at the level of whole datasets: scaling both sets together with the training set fitted keeps the arrangement of the points intact.

What Changes When the Test Set Scales Itself

What do you think happens?

If the scaler is fitted on the test set alone, where its minimum is 2 and its maximum is 8, what does the test value 2 become?

  • 0.0

  • 0.2

  • 2.0

  • 1.0

Reveal answer
Answer:

0.0

Under the test set's own minimum of 2 and maximum of 8, the value 2 is the minimum, so it maps to 0.0. The same input value that became 0.2 under the training rule now becomes 0.0. The value 8 is the maximum of the test set, so it becomes 1.0 instead of 0.8.

python
Output (expected)
[0. 1.]
Notes on the code
  • bad_scaler is a second, separate scaler built only from the test set
  • It records the test minimum of 2 and the test range of 6
  • The test values are stretched across the full 0 to 1 range
Test value 20.2Test value 20.0Test value 80.8Test value 81.0same inputsame input
How do the scaled test values differ when the test set uses its own minimum and maximum instead of the training minimum and maximum?

The third printed line is the misconception in action. The test values 2 and 8 have been stretched across the full range from 0.0 to 1.0, so the same input value now maps to a different output than it did a moment ago. Both sets now have a minimum of 0 and a maximum of 1, which looks tidy. But the test points have shifted relative to the training points because they were scaled differently. This rearranges the data in an arbitrary way, and that is not what we want.

The Line Where Results Diverge

When a scaling pipeline gives unexpected test results, the divergence almost always has a single location: the line where a scaler is fitted. Check each fit call and ask which dataset it received. If the fit call receives anything other than the training set, the transformed test set will be scaled by the wrong rules, even when every other line looks correct.

Code walkthrough
bad_scaler = MinMaxScaler()
bad_scaler.fit(X_test)  # divergence happens here
print(bad_scaler.transform(X_test).ravel())
  1. 1
    Line one creates a new MinMaxScaler, which is harmless on its own because nothing has been learned yet.
  2. 2
    Line two passes X_test to fit. This is the single line where the workflow goes wrong, because it creates a second, different set of rules from the test data.
  3. 3
    Line three applies those wrong rules to the test set. The output is correct arithmetic for the wrong rule, which is why the mistake is easy to miss.
  • The scaler is created correctly as a fresh object
  • The fit call on X_test is the divergence line
  • Everything after the fit inherits the wrong rules

Mistakes That Hide the Divergence

  • Common mistake

    Scaling the test set on its own minimum and maximum

    What it looks like

    Fitting a new MinMaxScaler on X_test and then transforming X_test with it, so the test value 2 becomes 0.0.

    Why it fails

    The test set gets its own rules, so the same input value no longer maps to the same output it would under the training rules.

    Do this instead

    Fit the scaler once on the training data, then transform both the training and test data with that same scaler.

  • Common mistake

    Fitting a scaler on the test data separately

    What it looks like

    Creating a second scaler just for the test set and calling fit on it, even when the first scaler was fitted on the training data correctly.

    Why it fails

    Calling fit on the test set creates a second, different set of rules, and the two sets are no longer read through one shared rule.

    Do this instead

    Call fit only on the training data, and reuse that fitted object for every transform call.

  • Common mistake

    Assuming a small test set can be scaled on its own

    What it looks like

    Believing that a test set with only one point can be scaled correctly by using that point's own minimum and maximum.

    Why it fails

    A single value has no spread of its own, so it cannot provide the minimum and range that MinMaxScaler needs. The size of the test set should never change the scaling method.

    Do this instead

    Derive the scaling method from the training data, regardless of how many test points exist.

  • Common mistake

    Judging a pipeline only by whether the output numbers look reasonable, rather than checking which dataset each fit call received

Checking Your Own Pipeline

MEDIUM

A colleague's workflow scales a training set with one scaler, then creates a second scaler, fits it on the test set, and transforms the test set with it. Identify the exact line where the test results stop matching what they should be, and describe what the correct version would do on that line instead.

Hints
  • Ask which dataset each fit call receives.
  • Only one call should learn the rules, and it should receive the training data.
  • The correct replacement reuses the already fitted scaler for the test set instead of fitting a new one.

Key Takeaways

  • In MinMaxScaler, fit learns the minimum and range from the training data, and transform applies those same parameters to the training set and the test set.
  • Fitting a scaler on the test set, or scaling each set on its own minimum and maximum, creates different rules, so the same input can map to a different output and test points can shift relative to training points.
  • The size of the test set should never change the scaling method, because a single test point cannot be scaled correctly on its own.
  • When test results are unexpected, check each fit call and ask which dataset it received; a fit call on the test data is the divergence point.

Related Topics

Related concepts

More concepts starting with T