Categorical Features

Imagine you have finished training a scikit-learn estimator on a set of examples. The next question is what you actually do with it. You might want the estimator to answer with a new output, such as a predicted label or a predicted number. You might instead want it to hand back a reworked version of the input data that is easier for another step to use. The estimator can do either, and the method you call decides which job it performs.

Two methods carry most of this work. The predict method creates a new output, such as a predicted label y. The transform method creates a new representation of the input data X. A summary of estimator types makes the choice concrete. Supervised and unsupervised tasks that produce an answer, such as classification, regression, and clustering, go through predict. Preprocessing, dimensionality reduction, feature extraction, and feature selection go through transform. If your goal is an answer, reach for predict. If your goal is a rewritten table of features, reach for transform.

This article follows one idea through both kinds of use and then applies it to a kind of input that needs special care: categorical features. Categorical features are the first case that needs attention in any representation scheme, because their lack of order means they cannot be fed to a model as if their numeric codes carried meaning on their own.

A concrete trace makes the difference clearer. The example uses a scaler, a preprocessing estimator whose transform method rescales features. Suppose the training data has two features in each row. You fit the scaler on the training rows, which lets it learn the statistics it needs. Then you call transform on a new row. The new row is never used to learn those statistics. It is only passed through them.

Notice the order of events. Fit happens once, on the training rows. Transform can then be applied to the training rows and to any new rows, and in both cases it reuses what fit already learned. The sequence below shows that ordering. The training data is used for fit, the training data and the new data both pass through transform, and the new data never changes what the encoder learned.

Shape deserves attention too. The shape of transformed data can stay the same, as in rescaling, or it can change when new features are added. Rescaling keeps one column per original feature, so the matrix you get back has the same number of rows and columns you started with. The one-hot encoding you will meet later is a case where the number of columns grows.

Everything in the scaler example used numbers that vary smoothly, such as a brightness or a length. Many real datasets do not look like that. A categorical feature, also called a discrete feature, describes which group an item belongs to. Its values are usually not numbers, and they do not form a continuous range. The brand, color, or department of a product are typical examples.

The difference lies in what sits between two values. A continuous feature can take any value in a range, and a value can sit between two others. A categorical feature offers no such middle ground. A product is either in the clothing department or in the books department. Books are not greater or less than clothing, and hardware does not sit between them. Categories have no natural order.

The contrast mirrors the difference between classification and regression, but on the input side instead of the output side. Its consequences matter for representation, because a model that works with numbers will treat whatever numbers you give it as if they had order and distance. That is the subject of the next section.

Sometimes a single continuous measurement is easier to model after it is chopped into intervals, called bins. The model then records only which bin each data point falls into. The result is no longer a measurement. It is a label: data point A is in bin 3, data point B is in bin 7, and so on. That label is a categorical feature.

A categorical feature is still a problem for a scikit-learn model, because the model needs numbers to do arithmetic on. Writing bin 7 as the number 7 would quietly tell the model that bin 7 is seven times bin 1, which is not what the bins mean. A linear model reads a larger code as a larger input value, so an integer code implies an order and a distance that do not exist between the groups.

One-hot encoding is the standard way out. It replaces each label with a vector of zeros and ones, one position per category, so that no category is treated as bigger or smaller than another. In the integer version, bin 7 looks far from bin 1. In the one-hot version, every bin is the same distance from every other bin, since each one is a separate column.

The following example uses a tiny feature of four data points so that you can check every value yourself. It is an original example written to follow the same fit and transform pattern as the source. The feature is named train_bins. It holds one bin label per row, and it is passed to the encoder as a two-dimensional array with one column.

Watch the two lines that touch the encoder. The fit line does not produce a matrix. The transform line is the one that produces the matrix you print. During fit, the encoder reads the feature and records which distinct values appear in it: here 1, 2 and 3. It stores that list and does not yet change any data. During transform, the encoder uses the stored list to build the matrix. For each data point, it places a 1 in the column belonging to that point's category and a 0 everywhere else.

Each row of the output is a binary vector. Exactly one entry in a row is 1, and the position of that 1 identifies the category of the data point. Fit decides the position: the categories are stored in sorted order, so bin 1 takes column 0, bin 2 takes column 1, and bin 3 takes column 2. The new data [2] and [1] is transformed with that same stored order, so bin 2 gives a 1 in column 1 and bin 1 gives a 1 in column 0. The encoder does not learn anything new from the new rows.

Choosing how to represent data is a step that happens before the model sees anything. The source makes a strong claim about it: how the data is represented can have an enormous effect on how well a machine learning model performs. Representing the data in the right way can influence a supervised model more than the exact parameters you choose. Finding the best representation for a particular application is called feature engineering.

Two representation decisions appear repeatedly in the source. The first is scaling. If the data is not rescaled, a measurement in centimeters and the same measurement in inches produce very different numbers for a model. The second is augmentation, where you add features derived from existing ones, such as interactions between features or more general polynomial terms. Both change what the model sees, and both can matter more than a parameter tweak.