Target Variable Transformation

Suppose you have a dataset, a model, and a pile of candidate transformations. Which one should you apply? The honest answer is that picking the right transformation is part judgment and part experiment. The best choice depends on the particular pairing of data and model, and nothing about one pairing guarantees the same result for another.

This article follows one question through the rest of the discussion: given a model, what should be changed in the data, and what should be left exactly as it is? Keep that question in mind while you read, because the answer changes with the model family. Tree-based models are less sensitive to transformations, while linear models often depend on them.

In a regression task, the target variable y is the quantity the model is trying to predict. Most of the transformation discussion concerns the features, which are the inputs the model sees. The source notes that it is sometimes worth transforming y as well. This is a different kind of change. Feature transformation alters what the model sees as input X. Target transformation alters what the model is asked to predict.

Once you know the model family matters, the next question is which features to touch. The source is clear that not all features require transformation. In a clean teaching example, every feature may share the same statistical properties, which makes one uniform treatment look natural. Real datasets rarely behave that way. Usually only a subset of the features needs transforming, and the remaining ones should be left as they are.

Even among the features that do need work, one treatment rarely fits all of them. Different features may call for different transformations. A feature whose values are spread across several orders of magnitude may need a different correction from a feature whose values already sit in a narrow band. For that reason, a sensible workflow inspects features one at a time, asks what each one looks like, and chooses a transformation for that feature specifically. The flowchart below sketches that sequence of checks.

The source singles out one common situation: predicting counts, such as the number of orders a customer places. Counting outcomes are a fairly routine regression task, and the source reports that applying the log(y + 1) transformation to the target often helps. The transformed target is $\log(y + 1)$, where $y$ is the original count.

Changing the target changes the order of operations. You do not feed the raw counts to the model. You transform y first, train on the transformed values, and the model's predictions come out in the transformed scale. To read a prediction as a count again, you must invert the transformation. Because $z = \log(y + 1)$ implies $y = e^{z} - 1$, a predicted value $\hat{y}$ in log space becomes $e^{\hat{y}} - 1$ on the count scale. The pipeline below traces that round trip.

Notice where the original count scale returns. It does not come back during training, and it does not come back at prediction time. It appears only at the last step, when the log-space output is inverted. A model evaluated only on log-space outputs can look fine while the counts it implies are off, so the inversion step belongs in your checks.

The +1 has two jobs. It keeps the transformation defined when the count is zero, and it compresses large values, since the logarithm grows slowly for big inputs. Both properties matter for count data, where zeros are common and a few very large counts can dominate a raw regression.

Consider a count target where most values are small but a few are very large. On the raw scale, the large values stretch the distribution into a long right tail. The table below shows several original counts next to their log(y + 1) values. The large counts move much closer together after the transformation, while zero stays a valid, finite value.