Representing Categorical Data for Machine Learning

A machine learning algorithm does not look at a spreadsheet the way a person does. It works with numbers arranged in a consistent layout, and the way those numbers are arranged can decide whether the algorithm learns anything useful. Real datasets often contain columns whose entries are words rather than numbers, and those columns need special handling before a model can use them.

A concrete case makes this easier to see. The adult income dataset is built from the 1994 census database in the United States. Its task is to predict whether a worker earns more than $50K per year or at most $50K per year. Each row describes one worker, and each column is a feature describing that worker: age, how they are employed, their education, their sex, their weekly working hours, their occupation, and more. The column we want to predict is income. The task is framed as a classification problem with two classes.

Read down the columns and notice a pattern. Some columns hold values that look like measurements, such as 39, 50, or 40. Other columns hold values that look like labels, such as Male, Bachelors, or Private. That difference in appearance is the starting point for the idea of a categorical variable.

The definition has two parts, and both matter. The first part is about where the values come from: a fixed list. The second part is about what the values mean: a quality rather than an amount. A categorical feature answers the question which one of these, not how much.

Apply the two questions to the adult dataset, one feature at a time. Workclass, education, sex, and occupation are the categorical features: their values come from a fixed set of labels and describe a quality of the worker. Age and hours-per-week are continuous features. They are numerical and their values fall along a range, so one value can be larger or smaller than another, and values can lie between two others.

The two groups look different on the page for a reason. Continuous values can be compared by size and can sit between two other values. Categorical values can only be matched or not matched against the list of labels. This difference will matter when we try to feed the columns to a model.

The reason categorical variables need attention appears once we train a model. Consider a logistic regression classifier on this data. A logistic regression makes its prediction by combining the input features with coefficients that are learned from the training set. The prediction is written as $\hat{y}$, and it depends on the coefficients $w_{i}$ and $b$ together with the input features $x_{i}$. The coefficients are fixed numbers learned during training, and the features are the values supplied for each example.

This weighted sum only works when the inputs are numbers. If $x_{2}$ is 39, multiplying it by a weight produces a meaningful quantity. If $x_{2}$ is the word Masters or Bachelors, there is no multiplication to perform. The formula makes sense when the input features are numbers, but not when a feature is a word, so the categorical values must be represented in some different way before logistic regression can be applied.

The standard way to convert categories into numbers is one-hot encoding. The idea is to give each possible category its own position in a vector. For a given data point, the position that belongs to its category is set to 1, and every other position is set to 0. Only one position is hot at a time, which is where the name comes from.

Suppose a dataset has a color column with three possible categories: red, green, and blue. First we fix the order of the categories, because the position of each 1 depends on that order. Then we write a small function that builds the vector for any single value. The function needs no library. It only compares the value against each category in turn.

Now apply the same function to every row of a small column. The same category always produces the same vector, so the two rows that say red come out identical. The single text column is replaced by a block of numeric columns, and each row becomes a vector.

Read the diagram by position. The slot labeled green is the only one set to 1, because green is the category of this data point. The vector length equals the number of categories, so the encoding grows wider as the number of categories grows.

Notice what this encoding avoids. If we simply assigned the numbers 0, 1, and 2 to red, green, and blue, a model could read those numbers as an ordering or as distances, even though the colors have no such order. A model that multiplies by a weight would treat green as lying halfway between red and blue.

Encoding changes the form of a feature, but feature engineering can also create new features from existing data. The chapter names three techniques: binning, adding polynomial terms, and adding interactions. Each reshapes a raw measurement into a form that gives a model more to work with. Expert knowledge of the problem usually guides which new features to create.

The engineered features do not replace the originals. They are added alongside them, and the combined set is what the model receives. The new columns are built from the same raw measurements, so the model sees both the plain values and the reshaped versions.

The right feature set depends on the model. A straight-line model cannot bend a relationship on its own, so linear models gain from binning and from adding polynomial and interaction terms. Those additions give a linear model the nonlinear structure it cannot produce by itself. More complex nonlinear models, such as random forests and SVMs, may learn complex tasks without the feature space being expanded explicitly.