Choosing Transformations for Linear Models

Count data shows up whenever you record how many times something happened: visits to a page, items bought in a store, or appearances of a word. Features like these tend to share a recognizable pattern. Most values are small, a few values are very large, and the bulk of the data sits on one side of the range.

A linear model fitted directly to a feature like this can struggle. A handful of extreme values pulls the fitted line toward themselves, so the model captures the main relationship poorly. In the example this article follows, a Ridge regression model achieved only a relatively low R squared score on the data, meaning it could not capture the relationship between the feature and the target well.

One way to address this is a logarithmic transformation of the feature. The rest of this article follows that transformation step by step: what it is, why it has a shift built into it, what it does to the shape of the data, and how to find the exact point where a pipeline starts producing a wrong result.

The transformation is a single line of NumPy applied to the training and test feature matrices. Before tracing it, make a prediction. Suppose one of the feature values in X_train is 0. What does the expression X_train + 1 produce for that entry, and why might that matter for the logarithm that follows? Write down your guess before reading on.

Compare your guess with the reveal. The point to hold onto is this: the problem is not the logarithm by itself. The problem is that a count of 0 is a legitimate value in count data, and the logarithm has no defined output for it. The shift is what resolves that conflict.

The source explains the reason for the shift directly. The value 0 appears in the data, and the logarithm is not defined at 0. So applying the logarithm to the raw values is not an option. Instead, the transformation is written as $\log(X + 1)$, which first adds 1 to each value and then takes the logarithm.

The shift guarantees that the argument of the logarithm is never zero for a count of zero, since 0 + 1 is 1. The original count of 0 is therefore mapped to a valid input rather than an undefined one. The shift also keeps every non-negative count within the domain of the logarithm, which is why the same expression works across the whole column.

Follow a single count column from raw input to the fitted model. The pipeline below has five stages. Trace where a bad value would first show up if the shift were left out.

Read the pipeline as a sequence of checkpoints. The raw counts still contain a 0, which is valid data. The unexpected result first appears at the logarithm stage if the shift is missing, because the logarithm of 0 has no valid value. Adding 1 at the second stage removes that problem before the logarithm sees it.

This example is a generated illustration that follows the pattern described in the source: one NumPy line applied to the feature matrix, with the shift applied before the logarithm. The zero in the first row maps to 0, and every other count maps to a finite value.

The mapping below shows how each raw count lands after the transformation. Look at the gap between neighboring values on each side. The raw counts jump by large amounts at the top of the range, while the transformed values move in much smaller steps.

The values in the map follow directly from the shift and the logarithm. Counts 0, 1, and 3 map to 0, about 0.693, and about 1.386. Counts 99 and 999 map to about 4.605 and about 6.908. Each step up by a factor of ten in the raw count adds the same amount, about 2.303, to the transformed value. That is why very large counts stop stretching the range the way they did before.

This is the compression the source describes: a logarithm compresses large values much more than small ones, so a long tail of big counts ends up closer to the bulk of the data.

A histogram shows how many values fall into each range, so it is a natural way to check whether the transformation reduced skew. The source plots the feature after the transformation, with a label on each axis and 25 bins. Compare that plot with a histogram of the raw column to judge whether the asymmetry and outliers have been reduced.

The source describes the effect in two parts. First, the distribution becomes less asymmetrical: the lopsided pile of small values and the long tail of large values become more balanced. Second, the very large outliers that stretched the range are no longer present in the same form. Both changes make it easier for a linear model to follow the main relationship, rather than being pulled by a few extreme points.

Check your answer by comparing it with the mapping from earlier. Counts 2 and 7 become log(3) and log(8), and 99 becomes log(100). If you are unsure about a value, return to the pipeline diagram and trace one count through each stage.