Model Evaluation with Train and Test Splits

Many real datasets contain a feature whose values are mostly small, with a handful of extremely large values sitting far to the right. Counts, prices, and durations often look like this. The feature may carry useful information about the target, yet a linear model can still struggle with it. This article shows why that happens, how a logarithm-based transformation helps, and how to decide which features are worth transforming.

Picture a feature whose values cluster near zero while a few values sit far above the rest. If you plot it, the points form a long tail stretching to the right. Statisticians call this a skewed distribution. Follow the values through a linear model step by step. A linear model fits a straight-line relationship between each feature and the target. When a few values are enormous, they dominate that fit. The many small values are squeezed into a tiny region of the scale, where the model cannot tell them apart well.

Consider a made-up count feature with five observations: 0, 1, 3, 12, and 950. Four of the values are below 13, but the last one is 950. On a raw scale, the gap between 0 and 12 is almost invisible next to the gap between 12 and 950. A straight line that tries to account for the 950 point is pulled far away from the cluster of small values, so the small values receive almost no distinct treatment. The fastest way to see this in practice is to fit a model on the raw data and look at the held-out test score.

A logarithm compresses large values. Once it is applied, the very large values move much closer to the bulk of the data, while the small values, which were crowded together, occupy a wider stretch of the scale. The feature becomes more evenly spread, and a linear relationship can describe it more effectively. The transformation changes the scale of the feature. It does not discard any of its values.

Read the diagram from top to bottom in each column. Before the transform, the raw gap from 12 to 950 is enormous compared with the gap from 0 to 3. After the transform, the largest value is 6.86, and the small values are spread across a visibly wider range relative to it. The 950 observation is still the largest value, but it no longer dominates the scale.

There is no single transformation that always works. Finding the best transformation for each combination of dataset and model is somewhat of an art. In the source's example, every feature happened to share the same properties, which is rarely true in practice. Usually only a subset of the features should be transformed, and sometimes each feature needs its own transformation.

The flowchart captures the main decision. Tree-based models are generally unaffected by these transforms, while linear models often depend on them. The source also notes that binning, polynomials, and interactions can have a large influence on less complex models such as linear models and naive Bayes models. Other models, including SVMs, nearest neighbors, and neural networks, may sometimes benefit, but the effect is usually less predictable. Because the effect is empirical, the practical test is to compare held-out scores with and without the transform.

Transformations change the values of existing features. A second lever is to change how many features the model sees. It is tempting to create many new features, but adding features makes all models more complex and raises the chance of overfitting. When the feature count is high, it can be wise to keep only the most useful features and discard the rest, which can produce simpler models that generalize better.

The source describes three basic strategies: univariate statistics, model-based selection, and iterative selection. All three are supervised, meaning they use the target. Univariate selection scores each feature independently against the target, keeps the top percentile of features, and passes the reduced set to the model. SelectPercentile is the scikit-learn tool for keeping a chosen percentage of the highest-scoring features.

Because feature selection uses the target, it is supervised, and it must be fitted only on the training portion of the data. The test set must remain untouched during selection. If the selector saw the test targets, the test score would no longer measure performance on unseen data, and it would look better than the model deserves.

Follow the diagram from the full dataset. The training split is used to fit the selector, which decides which columns survive. The test split then passes through the same learned selection without being used to make any choice. The model is trained on the reduced training data, and the test score reflects how it performs on rows it has never seen. Compare this score against the score from the unreduced features to judge whether the reduction helped.