Choosing Features for a Learning Method

Imagine two people given the same spreadsheet. One sees a column of city names. The other sees that same column already translated into a set of yes-or-no indicators, one per city. Both have the same facts, yet a learning method can only work with the second version, because it needs numbers in every cell. The question this article answers is how to move from raw columns to representations a learning method can use well, and why the features you choose should match the method you apply.

The representation of data should suit the learning algorithm. A categorical variable holds labels rather than quantities, such as a color or a product type. A continuous variable holds quantities that can take many values, such as a running number. Each kind of raw column has to be turned into something the method can consume. Two common moves do this work: one-hot encoding for categories, and binning for continuous values. Two further moves, polynomial and interaction terms, create new columns from existing ones. The rest of this article traces each of these moves step by step.

A categorical variable is replaced by several columns of zeros and ones, one column for each possible category. In each row, exactly one of those columns is set to 1, and it marks which category that row belongs to. The mechanism has two phases. First, you fix an ordered list of every category that appears. Second, for each value, you build a vector the same length as that list and place a 1 at the position of its category and 0 everywhere else.

Binning creates a new feature by grouping a continuous variable into bins. Instead of treating a value as a single running number, you assign it to the range it falls in. The result is a categorical-style feature that can then be encoded like any other category, which is why binning and one-hot encoding often appear together. The trade-off is that binning gives up fine detail in exchange for a clearer structure.

Another way to create new features is by adding polynomial and interaction terms. A polynomial term takes an existing feature and raises it to a power. An interaction term multiplies two different features together. Both kinds of term are new columns built from the original ones, and both give the model a way to express relationships that a single raw column cannot express on its own.

In this expression, x1 and x2 are the raw inputs. The terms $x_{1}^{2}$ and $x_{2}^{2}$ are polynomial terms, and the term $x_{1} x_{2}$ is an interaction term. Each one becomes its own column that the model weights, just like the original inputs.

A linear model combines its inputs with a weighted sum, so on its own it cannot bend a relationship. It can only express a relationship as a weighted combination of the columns it is given. Adding $x^{2}$ or $x_{1} x_{2}$ as columns lets the same weighted-sum machinery capture curvature or joint effects. The model is still linear in its weights, but because one of its columns is now a curved or combined quantity, the relationship it can express becomes richer.

The need to hand-build polynomial and binned features depends on the method you pair them with. Flexible nonlinear methods such as random forests and SVMs may learn complex tasks without explicitly expanding the feature space. The match between features and method is the deciding factor, not a single rule that applies everywhere.