One-Hot Encoding for Categorical Features

Many datasets are full of numbers that were never meant to be measured. A column might hold 0, 1, 2, and so on, while the thing it describes is a set of named options such as a job type, a country, or a survey answer. Once the labels have been replaced by digits, the table looks like it contains quantities, and it is easy to forget what the digits stand for.

This article follows one question through to the end: how do you decide what kind of feature an integer column really is, and what has to happen before a model can use it? The answer begins with a common assumption that is worth examining first, because it is the one most likely to cause trouble.

A common assumption goes like this: if a column contains integers, then its values are counts or measurements, so the column is continuous and the numbers can be compared, averaged, and fed to a model as ordinary magnitudes. This belief treats the data type of the stored value as if it revealed the meaning of the feature.

Categorical data is often stored as integers for practical reasons: integers are easier to store, or the way the data was collected produced numeric codes. In the adult dataset, a variable such as workclass was stored as text strings like Private. Text clearly marks a variable as categorical, but it has a drawback, because spelling errors become possible. Storing the same information as integers avoids spelling errors but removes the visible signal that the column is categorical.

What resolves the ambiguity is knowing what the numbers mean. Once you know that the codes indicate employment status, it becomes clear that these are distinct states that should not be modeled by a single continuous variable. The integer type is a fact about storage. The category is a fact about meaning, and only the meaning should guide how the feature is modeled.

Imagine a single continuous measurement, such as a number that can take any value along a range. Sometimes it helps a model to chop that range into intervals, called bins, and then record only which bin each data point falls into. The result is no longer a measurement. It is a label: data point A is in bin 3, data point B is in bin 7, and so on. That label is a categorical feature.

The central question for any integer feature is whether the encoded values have an ordering. Ordering means that moving from one code to another corresponds to a real, meaningful step along some scale. When no such ordering exists between the encoded semantics, the feature must be treated as discrete, and in practice it is typically one-hot encoded so that each state gets its own indicator.

Work through the checks in order. First ask whether the integers are real amounts, such as counts or measurements, where a difference of 2 means the same kind of gap anywhere on the scale. If they are, the column is a quantity and can usually stay as a single numeric column. If they are not, ask whether the codes name options that lie along a meaningful scale. If they do, the column is an ordered category. If the codes name options with no scale between them, the column is an unordered category and should be treated as discrete. The workclass example falls into the last group, since its codes name employment statuses that do not lie along any scale.

Not every integer feature is clearly categorical. A five-star rating has a natural order: four stars is higher than two. That ordering means the feature is not simply a set of unrelated labels. Even so, the choice is not fixed. For a case like five-star ratings, the better encoding depends on the particular task and data, and on which machine learning algorithm is used. There is no single correct answer that applies everywhere. So an ordered integer column is a case where you must make a context-dependent decision rather than apply a rule mechanically.

A categorical feature is still a problem for a scikit-learn model, because the model needs numbers to do arithmetic on. Writing bin 7 as the number 7 would quietly tell the model that bin 7 is seven times bin 1, which is not what the bins mean. A single integer column implies a numeric ordering and spacing: the gap between codes 1 and 2 looks the same as the gap between 2 and 3, and code 7 looks like seven times code 1. A linear model would use those implied distances, even though the categories have no such distances.

One-hot encoding is the standard way out. It replaces each label with a vector of zeros and ones, one position per category, so that no category is treated as bigger or smaller than another. Each category sits in its own column, and every pair of different categories is equally different from the model's point of view.

OneHotEncoder works in two phases. During fit, the encoder reads the feature and records which distinct values appear in it. It stores that list of categories and does not yet change any data. During transform, the encoder uses the stored list to build the matrix: for each data point, it places a 1 in the column belonging to that point's category and a 0 everywhere else. Keeping these two phases apart is the key to debugging the encoder.

The number of output columns equals the number of unique categories. This is why a feature with 10 specified bins produces 10 features. Before you look at the printed matrix, you can work out its width from the labels alone: count the distinct values, and that count is the number of columns.

In this expression, k is the number of output columns and C is the set of unique categories recorded during fit. Here the symbol for set size means the count of distinct labels.