Working with Categorical Data

Many datasets describe people, products, or events with labels instead of numbers. A column might record a job type, a color, or a region. A learning algorithm cannot do arithmetic on a label such as 'Private Employee' in any meaningful way, so the label has to be translated into a form the model can use. This article follows one common translation, one-hot encoding, and asks a single question: how does a column of labels become a set of numeric features, and what does each row look like afterward?

A linear model combines feature values with weights. Each feature value is multiplied by its weight, and the products are added together to produce the model's output. This works well when every feature is a number. It does not work when a feature holds text. If a row's color is the text 'Blue', there is no weight that can be multiplied by the word 'Blue' to produce a meaningful contribution to the output. The model needs numbers in every position it uses.

The solution is to rewrite the label as numbers that the model can weight. The rewriting has to preserve the information in the label: a row from one category must look different from a row from another category. One-hot encoding does this by giving each category its own feature, so that each category can contribute its own weight to the result.

The method is built around a simple rule. One new feature is created for each category. Each new feature answers a yes-or-no question about a single data point: does this row belong to this category? The feature is 1 if the data point has the corresponding category, and 0 otherwise. Because every category gets its own feature, the scheme can represent any number of categories, with one feature per category.

The name 'one-hot' comes from what this rule produces. For any single data point, exactly one of the new features is 1 and the rest are 0. Only one position is 'hot' at a time, which is why the method is also described as one-out-of-N.

Once the new features exist, the original categorical column is no longer needed. When the data is used in a machine learning algorithm, the original column is dropped and only the 0-1 features are kept.

The running example uses a feature called workclass with four possible values: 'Government Employee', 'Private Employee', 'Self Employed', and 'Self Employed Incorporated'. Encoding these four values creates four new features, named after the same four categories. Each row below shows one data point, with the single 1 marking its category.

Now compare that row with a row whose workclass is 'Self Employed'. The four positions are the same, but the 1 moves to the 'Self Employed' position. Every row has the same four columns, and the pattern of 1s and 0s is what tells the model which category the row belongs to.

The idea can be written directly in Python without any special library. The function below takes a value and the list of known categories, then returns a list with a 1 in the position of the matching category and 0 everywhere else. This is a generated illustration of the same rule described above, not a library call.

The output confirms the rule. The first row has exactly one 1, in the second position, which corresponds to Private Employee. The second row has its 1 in the fourth position. The category order in the categories list must stay the same for every row; otherwise the same position would stand for different categories in different rows.

The one-hot encoding used in machine learning is quite similar to the dummy encoding used in statistics, but the two are not identical. The machine learning version gives each category its own binary feature. The statistics version differs in one specific way: dummy encoding uses $k-1$ features for $k$ categories, and the last category is represented as all zeros.

The reason given in the source is analytical. Using $k-1$ features simplifies the analysis, and more technically, it avoids making the data matrix rank-deficient. The source does not go further into the rank argument, so treat that as the stated purpose rather than a full derivation.

To see how the dropped category is still recognized, consider a generated example with three colors: Red, Green, and Blue, treated in that order. Dummy encoding keeps a column for Red and a column for Green. A row that is Red gets [1, 0]. A row that is Green gets [0, 1]. A row that is Blue gets [0, 0]. The all-zero pattern stands for the last category, Blue, which has no column of its own.