Handling Categorical Variables

Imagine two people given the same spreadsheet. One sees a column of city names. The other sees that same column already translated into a set of yes-or-no indicators, one per city. Both people have the same facts, yet a learning method can only work with the second version, because it needs numbers in every cell. This article follows that translation step by step: how a label becomes a row of zeros and ones, how a continuous number becomes a bin label, and how new columns can be built from existing ones.

The question at the center of this article is how to move from raw columns to representations a learning method can use well, and what it means for the features you choose to match the method you apply. Feature engineering is the process of creating new features from existing ones so that a model has better material to learn from. The features chosen, and how well they fit the machine learning method, often matter more to the final result than any other single decision in the pipeline.

The mechanism has two phases. In the first phase you fix an ordered list of every category that appears in the data. In the second phase, for each value, you build a vector the same length as that list and place a 1 at the position of its category and a 0 everywhere else. Because each category gets its own position, the encoding does not force any ordering on the categories that the original labels did not have. The label red is not treated as smaller or larger than green; it simply occupies its own column.

A learning method can only compute with numbers in every cell. A raw text label such as red gives a linear model no quantity to multiply by a weight, so the model cannot draw a useful boundary from it directly. Once the same information is expressed as encoded numeric columns, the weighted sum has something to work with. The facts did not change between the two versions of the spreadsheet; only the representation did.

The source also makes a point about method choice. Flexible nonlinear models such as random forests and SVMs may learn complex tasks without explicitly expanding the feature space. So whether you need to hand-build polynomial or binned features depends on the method you pair them with. The match between features and method is the deciding factor.

Binning creates new features by grouping a continuous variable into bins. Instead of treating age as a single running number, you assign each value to the range it falls in, such as child, adult, or senior. The result is a categorical-style feature that can be encoded like any other category. This is why binning and one-hot encoding often appear together. Binning trades fine detail for a clearer structure: two ages in the same bin become indistinguishable to the model.

Another way to create new features is by adding polynomial and interaction terms. A polynomial term takes an existing feature and raises it to a power, such as $x^{2}$. An interaction term multiplies two different features together, such as $x_1 x_2$. Both kinds of term are new columns built from the original ones, and both give the model a way to express relationships that a single raw column cannot express on its own.

Why does this help a linear model? A linear model combines its inputs with a weighted sum, so on its own it cannot bend a relationship. Adding x squared or x1 times x2 as columns lets the same weighted-sum machinery capture curvature or joint effects. The model still only adds weighted columns; the new columns are what carry the curvature or the joint effect.