Tree-Based Models and Feature Interactions

Many real datasets contain a feature whose values are mostly small, with a handful of extremely large values sitting far to the right. Counts, prices, and durations often look like this. The feature may carry useful information about the target, yet a linear model can still struggle with it. This article follows one feature from its raw form through a transformation and a feature-selection step, and shows how to check each change with a score on unseen data.

Before reading on, make a prediction. Suppose you fit a ridge regression on raw columns where most values are small but a few are enormous. Will the model fit well, or will the large values get in the way? Keep your answer in mind, because the rest of the article explains what happens and why.

Picture a feature whose values cluster near zero, while a few values sit far above the rest. Plotted on a number line, the feature shows a long tail stretching to the right. Statisticians call this a skewed distribution. The source describes this pattern as many small values together with a few very large ones, and notes that it is very common in practice.

A linear model fits a straight-line relationship between each feature and the target. When a few values are enormous, they dominate that fit. The many small values are squeezed into a tiny region of the scale, where the model cannot distinguish them well. The diagram below contrasts the raw feature with the same feature after a logarithm is applied.

The problem is not that the large values are wrong. They are valid observations, but on a linear scale they carry so much weight that the model's line is set mostly by them. A transformation that compresses large values makes the feature easier for a linear relationship to describe.

The source uses the form log(x + 1) when it transforms a feature or a target that counts things. The logarithm compresses large values, so very large numbers move much closer to the bulk of the data. Meanwhile, the small values, which were crowded together, occupy a wider stretch of the scale. The feature becomes more evenly spread. The transformation changes the scale of the feature rather than discarding any of its values.

Adding 1 matters because many count values are zero, and the logarithm of zero is undefined. Adding 1 moves every value of zero or more into a range where the logarithm is defined. This is why the form log(x + 1), and the NumPy function np.log1p that computes it, is the usual choice for counts and for regression targets.

There is no single transformation that always works. The source notes that finding the best transformation for each combination of dataset and model is somewhat of an art. Which features to transform is an empirical choice. In the source's example every feature happened to share the same properties, which is rarely true in practice. Usually only a subset of the features should be transformed, and sometimes each feature needs its own transformation.

The choice depends on the model family. Tree-based models are generally unaffected by these transforms, while linear models often depend on them. The source also says that binning, polynomials, and interactions can have a large influence on less complex models such as linear models and naive Bayes models. Other models, including SVMs, nearest neighbors, and neural networks, may sometimes benefit, but the effect is usually less predictable.

A tree splits the feature at thresholds, so it separates small values from large ones without needing the values to lie on a straight-line relationship. Because of that, the compression a logarithm provides usually gives a tree little benefit. This is the reason tree-based models usually skip this step, while a linear model often needs it.

Transformations change the values of existing features. A second lever is to change how many features the model sees. The source warns that it is tempting to create many new features, but adding features makes all models more complex and raises the chance of overfitting. When the feature count is high, it can be wise to keep only the most useful features and discard the rest. This can produce simpler models that generalize better.

The source describes three basic strategies: univariate statistics, model-based selection, and iterative selection. This article focuses on the first. Univariate selection scores each feature on its own against the target, then keeps the top-scoring features. SelectPercentile keeps a chosen percentage of the features with the highest scores.

Feature selection is supervised: it uses the target to decide which features to keep. For that reason the selection must be fitted only on the training portion of the data, and the test set must remain untouched. The data flows in one direction. The full dataset is split into a training set and a test set. The selector and the model are fitted on the training set. The test set is then transformed using the already-fitted selector and scored once.