Representing Categorical Variables

A machine learning algorithm does not look at a spreadsheet the way a person does. It works with numbers arranged in a consistent layout, and the way those numbers are arranged can decide whether the algorithm learns anything useful at all. Many real datasets contain columns that hold categories rather than measurements: a color such as red, green, or blue, a city name, or a product type. Those columns cannot be fed to an algorithm as they are, because the algorithm's arithmetic needs numbers.

This article follows one idea. The data you feed a model must be represented in a form that suits the algorithm, and the features you choose must match the method you plan to use. The source makes a practical point that is easy to underestimate: the features that are used, and how well they match the chosen method, is often the most important factor in whether a machine learning approach works well. Before worrying about which algorithm is the most sophisticated, it helps to ask whether the inputs are shaped the right way.

The standard way to convert a categorical column into numbers is one-hot encoding. The idea is to give each possible category its own position in a vector. For a given data point, the position that belongs to its category is set to 1, and every other position is set to 0. Only one position is hot at a time, which is where the name comes from.

Before any encoding can happen, you fix the order of the categories, because the position of each 1 depends on that order. Suppose a dataset has a color column with three possible categories, listed in the order red, green, blue. The vector for any value therefore has three positions: the first belongs to red, the second to green, and the third to blue. A red value puts its 1 in the first position, a green value in the second, and a blue value in the third.

The diagram above shows the vector for a red value. Each position is labeled with the category it belongs to, so reading the vector back is a lookup: the position holding the 1 names the category. The vector length equals the number of categories, so this encoding grows wider as the number of categories grows.

The encoding does not need any library. A small function can build the vector for any single value by comparing that value against each category in turn. The code below does exactly that, and then applies the same function to every row of a short column.

Read the output rows against the category order. The first and last rows are both red, and both carry a 1 in the first position, which is why they match. The single text column has been replaced by a block of numeric columns, and each row is now a vector the algorithm can use.

A tempting shortcut is to assign each category a single number: red becomes 0, green becomes 1, and blue becomes 2. This produces one column instead of three, but it introduces a problem. If you simply assign the numbers 0, 1, and 2 to red, green, and blue, the model could read those numbers as an ordering or as distances, even though the colors have no such order. A model that sees blue as 2 and red as 0 may treat blue as being twice as far from the start as green is.

One-hot encoding avoids this. It keeps each category separate, so no category is numerically treated as lying between two others. In the one-hot vectors, every pair of different categories differs in exactly two positions, so no pair looks closer than another. The model is not told that green sits between red and blue.

Feature engineering creates new features from existing data, and the chapter names three techniques: binning, adding polynomials, and adding interactions. Binning takes a continuous measurement and converts it into interval-based categories. Each interval becomes an indicator column that is 1 when the value falls inside it and 0 otherwise. The result is a set of bin indicators in place of one raw number.

The bin boundaries in this example were chosen only for illustration. In a real project they would come from knowledge of the problem, which is the kind of expert input the source describes. Choosing boundaries that have no real meaning for the problem gives the model intervals that carry no useful information.

A polynomial term is a new column built from one existing feature by raising it to a power, such as the square of $x_{1}$, written $x_{1}^{2}$. An interaction feature is a new column built by multiplying two existing features together, such as $x_{1} x_{2}$. Both techniques create columns that did not exist in the raw data, and both are derived entirely from columns that did.

The diagram shows that the squared term draws only on x1, while the interaction term draws on both x1 and x2. The engineered columns are added alongside the originals, and the combined set is what the model receives. The originals are not discarded.

The chapter's central warning is that the right feature set depends on the model. Feature choices are not universal. A feature that rescues one kind of model may add nothing to another, which is why the match between the chosen features and the chosen method is often the most important factor in making a machine learning approach work well.

The practical rule that follows is to ask which model you plan to use before deciding how much feature engineering to do. A linear model is a good candidate for explicit binning, polynomial terms, and interactions. A more complex nonlinear model may already learn those patterns, so expanding the feature space by hand may add effort without adding benefit.

Most errors in this topic come from treating the encoding as a simple relabeling. The following mistakes come up often when learners first convert categories into numbers.