Transforming Features for Linear Models

Imagine you have trained a Ridge regression model and checked its R 2 score. The score is relatively low, which means the model was not able to capture the relationship between the feature X and the target y. The natural reaction is to suspect the model, but a poor score is often telling you something about the shape of the data itself.

The fix introduced in this part of the chapter does not change the model at all. It changes the feature values before the model sees them. Transforming a feature so that a linear model can use it more effectively is the central idea of this chapter, and the transformation studied here is the logarithm. As you read, keep one question in mind: what does this line of code do to each value in the array, and why would a model respond differently to the result?

The feature in this example contains the value 0. The logarithm is not defined at 0, so the transformation cannot simply be the logarithm of each value X. If you applied the plain logarithm to a zero, the result would not be a usable number, and the transformed feature would be broken before the model ever saw it.

The source solves the zero problem by adding one before taking the logarithm. Each value in the feature is replaced by the logarithm of that value plus one. When X is zero or positive, the input to the logarithm is at least 1, so it is always strictly greater than zero and the logarithm is defined.

In Python this is written as np.log(X + 1). NumPy applies the expression to every element of an array at once, so one line of code transforms an entire feature column. The +1 is applied first, inside the parentheses, and the logarithm is taken of the result.

The diagram below traces a few generated input values through log(X + 1). Notice that the input 0 fails under the plain logarithm but produces a valid output once one is added first. Each arrow shows the value entering the transform and the value that comes out.

The purpose of the transformation is to make the data more symmetrical and to reduce the effect of outliers. Before the transformation, the feature is asymmetrical: most of its values sit in one region, and a few very large values stand apart from the rest. This is the right-skewed shape shown on the left of the diagram below. After the transformation, the distribution is less asymmetrical and the very large outliers are no longer present in the same way, as shown on the right.

The compression is the mechanism behind the change. Because the logarithm stretches small values apart and squeezes large values together, the long tail of big numbers is pulled inward. The values that were far apart on the original scale end up much closer together, which is why the transformed feature is easier for a linear model to use.

The source applies the transformation to both the training split and the test split, producing two new arrays. The same expression is used for each, so the model sees values that were produced in the same way whether it is learning or being evaluated. The code below is a generated illustration of that pattern, using the array names the source uses for the transformed data.

A histogram groups values into bins and shows how many values fall into each bin, which makes it the natural way to see whether the distribution has become more symmetrical. The source plots the first column of the transformed training data with 25 bins in gray and labels its axes. The code below is a generated version of that plotting step.