Train-Test Splitting in Supervised Learning

Every machine learning experiment needs a starting point. Before you can say that a change helped, you need a number that the change is measured against. In supervised learning, that starting point is usually a model trained on the data as it is, with no changes to the input values, and then scored on examples it has never seen. This article uses exactly that kind of experiment. A Support Vector Machine, a supervised learning classifier, is trained on a breast cancer dataset that has been split into a training set and a test set. The model is first fitted on the raw feature values and scored on the raw test set. The score from that first run is the baseline, and every later result in the experiment is only meaningful as a comparison to it.

Start by reading the baseline run line by line, because the rest of the exercise depends on it. The dataset is split into training features and test features. The training data is the only data the model learns from. The test data is held back and used only to check how well the learned model works on examples it did not see during fitting. The SVC is fitted on the raw training features and then scored on the raw test features.

The number 0.63 is the fraction of test examples that the unscaled model classified correctly. It is not a measure of the training data, and it is not a measure of how the model would behave on new data beyond this split. It is simply the share of held-out examples the model got right under the raw feature values. Because this number is the reference point, reading it carefully matters more than memorizing it.

The scaled experiment introduces a new object, a scaler, and three calls on it. The fit call is where the scaler learns the parameters it needs from the data it is given. The transform call applies those learned parameters to a dataset. The two calls do different jobs. Fitting looks at data and records something about it. Transforming uses what was recorded to change values. In the source code, fit is called only on the training features, and the test features are passed only to transform.

The key distinction is between fitting the scaler and transforming with it. The scaler is fitted once, from the training features. That single fitted scaler then produces both scaled datasets. The test features are transformed with the parameters learned from training data, not with parameters learned from the test features themselves. If you called fit on the test features, or created a new scaler for the test set, the test set would be scaled according to its own values rather than the training values, and the test score would no longer be produced under the same conditions as the training.

Now trace the scaled experiment in order. The source runs the SVC on scaled features after the same split used in the baseline. The preprocessing is inserted between the split and the model fit, so the model is never shown the raw training values in the scaled run. Each step has a definite input and output, and the final accuracy depends on all of them.

The jump from 0.63 to 0.97 is the size of the change produced by preparing the features. The source does not state a general law that scaling always helps an SVC. It reports the difference in this one experiment, and it is the comparison between the two printed lines that carries the result. What the experiment does show is that the decision boundary the SVC learns depends on the values it is given, so changing how those values are prepared changes the boundary and therefore the predictions on held-out data.

The exercise also runs the same experiment with StandardScaler. The lines use the same fit, transform, and transform pattern, and only the scaler class changes. The two classes do different things to the feature values, so it is worth separating them clearly. MinMaxScaler scales data to a 0 to 1 range. StandardScaler scales data to have zero mean and unit variance. Both are applied in the same place in the workflow: after the split, fitted on training features, and used to transform both sets before the SVC is trained and scored.

Suppose you run the scaled experiment and the accuracy comes out near the baseline instead of near 0.97. The source's advice is to trace the data through the code in order and ask, at each line, which object and which values are now in play. Most surprising results in this kind of experiment trace back to one of a small number of mistakes in the fit, transform, and score calls.