Using pandas get_dummies for Categorical Data
A binned continuous feature becomes a categorical feature, and one-hot encoding converts that categorical feature into numbers a scikit-learn model can use.
Why Bin Labels Are Not Plain Numbers
Imagine a single continuous measurement, such as a number that can take any value along a range. Sometimes it helps a model to chop that range into intervals, called bins, and then record only which bin each data point falls into. The result is no longer a measurement. It is a label: data point A is in bin 3, data point B is in bin 7, and so on. That label is a categorical feature.
What do you think happens?
If you write bin 7 as the number 7 and feed it to a model, what does the model conclude about bin 7 compared with bin 1?
Reveal answer
That bin 7 is seven times bin 1
A linear model performs arithmetic on the numbers it receives, so writing bin 7 as 7 quietly tells it that bin 7 is seven times bin 1. That is not what the bins mean, because the bins are just different categories.
A categorical feature is still a problem for a scikit-learn model, because the model needs numbers to do arithmetic on. One-hot encoding is the standard way out. It replaces each label with a vector of zeros and ones, one position per category, so that no category is treated as bigger or smaller than another. The rest of this article follows the source's approach: run the encoder on binned data first, then work out what each step did.
Tracing the Encoder's Two Phases
The encoder works in two phases, and keeping them apart is the key to debugging it. During fit, the encoder reads the feature and records which distinct values appear in it. It stores that list of categories and does not yet change any data. During transform, the encoder uses the stored list to build the matrix: for each data point, it places a 1 in the column belonging to that point's category and a 0 everywhere else.
[array([1, 3, 7])]
[[0. 1. 0.]
[1. 0. 0.]
[0. 0. 1.]
[0. 1. 0.]]- The feature is a column of integer bin labels, which is the form OneHotEncoder accepts.
- fit learns the distinct categories 1, 3, and 7 and stores them in categories_.
- transform uses the stored list to build the matrix; the printed matrix is the output of this line, not of fit.
Notice the two separate calls. The fit line produces no matrix; it only records the categories. The transform line is the one that produces the matrix you print. The stored list printed by categories_ is the vocabulary that transform reuses. When transform sees a label of 3, it looks up where 3 sits in that stored list and places the 1 there.
Read the diagram from the top down. The stored list fixes the order: category 1 owns position 0, category 3 owns position 1, and category 7 owns position 2. Fit is what determines this ordering, because it is the step that records the categories. For a row whose label is 3, only the middle cell is set to 1, and the other two stay 0. Each row of the output is a binary vector with exactly one 1, and the position of that 1 identifies the category.
Following Four Data Points Through the Encoder
The following example uses a tiny feature of four data points so that you can check every value yourself. It is an original example written to follow the same fit and transform pattern as the source code.
Encoding the labels 3, 1, 7, 3
A feature holds the bin labels 3, 1, 7, and 3 for four data points. Fit an OneHotEncoder on this feature and then transform it. Find the stored categories, the position each category maps to, and the final matrix.
- 1Fit: record the distinct values
The distinct values in the feature are 1, 3, and 7. fit stores this list in the encoder and does not change any data yet.
- 2Assign one position per category
The stored list gives the order: 1 goes to position 0, 3 goes to position 1, and 7 goes to position 2. The matrix will therefore have three columns.
- 3Transform the first row
The first data point has label 3. Its category sits at position 1, so the row is 0, 1, 0.
- 4Transform the remaining rows
Label 1 sits at position 0, giving 1, 0, 0. Label 7 sits at position 2, giving 0, 0, 1. The fourth label is 3 again, so its row is 0, 1, 0, the same as the first.
- 5Read the columns
Column 0 holds 0, 1, 0, 0, which marks the data points labelled 1. Column 1 holds 1, 0, 0, 1, which marks the points labelled 3. Column 2 holds 0, 0, 1, 0, which marks the point labelled 7.
Four rows and three columns. Each row has exactly one 1, and the two rows labelled 3 are identical.
Before you move on, check the logic against the result. The two rows that share label 3 must match, because transform looks up the same stored position both times. If they did not match, the fault would lie in the transform step, not in the data.
Counting Output Columns
The number of output columns equals the number of unique categories. If a feature has $k$ unique categories, one-hot encoding turns that single column into $k$ binary columns, and the number of rows stays the same. This is why 10 specified bins produce 10 features.
In the before picture the feature is one column with four rows. In the after picture the same four rows are kept, but the single column has been replaced by three columns, one for each unique category. The row count does not change; the column count becomes the number of categories.
| Feature | Integer-coded column | One-hot encoded columns |
|---|---|---|
| Representation of bin 7 | A single number 7 | A vector with a 1 in the bin 7 position and 0 elsewhere |
| Implied relationship | Bin 7 is seven times bin 1, and sits a fixed distance from bin 3 | No category is bigger or smaller than another |
| Number of columns | One | One per unique category |
| Suitable for a linear model's arithmetic | Misleading, because the arithmetic follows the numbers | Each category gets its own column, so no false ordering is introduced |
Why the integer-coded column misleads a linear model and the one-hot columns do not
Where a Wrong Result Begins
When an encoded matrix looks wrong, the most useful question is where the divergence starts. Because the work splits into fit and transform, you can check the stored categories first and the matrix second. The diagram below shows the path from a raw label to a matrix row, and the decision point where an unexpected value first shows up.
The decision node is the place to look. A label that is in the stored list gets a position and therefore a 1 in its column. A label that is not in the stored list has no column to receive the 1, so the row cannot be built the way it was for the training categories. The divergence therefore begins at the moment transform checks a value against the list that fit recorded, not at the matrix you print.
Common mistake Feeding the bin number as a plain number and assuming the model understands it as a category
What it looks likeUsing 7 as the value for bin 7 and 1 as the value for bin 1
Why it failsThe model does arithmetic on the numbers, so it treats bin 7 as seven times bin 1.
Do this insteadEncode the labels with one-hot encoding so each category gets its own column.
Common mistake Expecting one column per data point instead of one column per category
What it looks likeFour data points with three unique labels producing four columns
Why it failsThe number of output columns equals the number of unique categories, not the number of rows.
Do this insteadCount the unique categories first. Three unique labels give three columns.
Common mistake Looking for the fault in the printed matrix before checking the stored categories
What it looks likeRe-running the transform repeatedly when the categories list is already wrong
Why it failsThe matrix is built from the list fit stored, so a wrong list makes every row wrong.
Do this insteadPrint the stored categories first, then check the matrix against them.
Common mistake Passing a label that was absent from the training data and expecting it to get a column
Practice With a New Feature
A feature holds the bin labels 2, 2, 5, 9, and 5 for five data points. Before running any code, predict how many output columns one-hot encoding will produce. Then state which binary row the data point with label 9 receives, using the stored categories in ascending order.
Hints
- Count the distinct values first, not the number of rows.
- fit stores the categories in ascending order: 2, 5, 9.
- Label 9 is at position 2 in that list, so its row has a 1 only in the last column.
Summary of the Encoding Steps
- A binned continuous feature becomes a categorical label, and a model needs numbers, so the label must be encoded.
- fit records the distinct categories and changes no data; transform builds one binary row per data point using that stored list.
- The number of output columns equals the number of unique categories, so $k$ categories give $k$ columns.
- When the output is wrong, check that the input is integer-coded, then check the stored categories, then check the matrix rows.
At a glance
- A categorical feature must be converted to numbers before a linear model can use it, because the model would otherwise treat category labels as ordered quantities.
- OneHotEncoder's fit step records the distinct categories, and its transform step places a 1 in the column for each row's category and 0 elsewhere.
- The number of output columns equals the number of unique categories, so the column count is predictable before any encoding runs.
- When an encoded result is wrong, check that the input is integer-coded first, then the stored categories from fit, then the matrix from transform.