Encoding Categorical Data
When a categorical feature is stored as integer codes, the model reads those codes as continuous quantities.
Why Numbering Categories Feels Harmless
A very common first attempt at feeding categories to a model is to number them. Day of week becomes 0 through 6, or time of day becomes a count of hours. The numbers look harmless, and a linear model accepts them without complaint. The belief behind this shortcut is that a number is a number, so the model will treat it correctly. This article examines why that belief fails for categorical variables, and how one-hot encoding repairs it.
What do you think happens?
Suppose day of week is stored as 0 for Monday up to 6 for Sunday and the model uses that column as one input. Is the predicted change when the code moves from 0 to 1 different from the predicted change when it moves from 5 to 6?
Reveal answer
No, the change is the same at every step.
A linear model with one integer-coded input predicts y-hat = w x + b. Moving x from 0 to 1 changes the prediction by w, and moving from 5 to 6 also changes it by w. The single coefficient w is shared by every step along the code, so the model cannot give Monday-to-Tuesday a different effect from Saturday-to-Sunday.
Why Integer Codes Behave Like a Ruler
When a categorical feature is stored as integer codes, the model reads those codes as continuous quantities. In a linear model, the input contributes to the prediction through a single learned coefficient. The form of the prediction is shown below, where x is the integer code, w is the single learned coefficient, b is the intercept, and y-hat is the predicted value.
Every category shares the same w. The effect of the category is therefore forced onto a straight line through the codes. Each step from one code to the next adds the same amount to the prediction, no matter which categories those codes stand for. The diagram below shows three colors encoded as red equals 0, green equals 1, and blue equals 2. The model has one slope, so the step from red to green is the same size as the step from green to blue, and green's effect is forced to sit exactly halfway between red's and blue's.
Bike Rental Data and the Straight-Line Failure
The source makes this point with a bike rental example. Day of week and time of day were encoded as integers and then treated as numeric inputs. The model's predictions came out much worse, and the periodic pattern in the data looked wrong. The explanation given is that integer codes are interpreted as continuous quantities, so a linear model can only learn a linear function of time of day. Under that function, later hours always push the prediction in one direction, and the model concluded that later in the day there are more rentals.
The real pattern in the data is more complicated than a steady rise or fall across the day. Rentals can rise and fall in ways that repeat, so no single straight line through the hour codes can match them. The issue is not that the model is poorly fit. The issue is that the input representation forces the model to use one slope for everything.
How One-Hot Encoding Changes the Input
One-hot encoding creates one indicator column per category. Each row gets a 1 in the column for its own category and 0 in every other column. Instead of one input that carries all the categories on a single scale, the model now receives several inputs, and each input answers a yes-or-no question about one category.
In the diagram, one categorical entry enters the encoder and leaves as several columns. Exactly one of those columns carries the 1, and that is the column whose coefficient will multiply for that category. The other columns are 0 and contribute nothing for that row.
The effect on the model is the important part. Once the categories are indicator columns, the linear model learns one coefficient for each category. Each category gets its own adjustment, so the model is no longer forced to draw a straight line through the codes. The source reports that this gives a much better match to the bike rental data than the continuous encoding did.
Encoding One Row by Hand
Encoding the color green
A categorical feature called color has three possible values, red, green, and blue. A row has color equal to green. What indicator values does one-hot encoding produce for that row, in the order blue, green, red?
- 1List the categories
The feature has three categories, so the encoding produces three indicator columns.
- 2Find the row's own category
The row's category is green, so the green column is the one that receives a 1.
- 3Fill the other columns with 0
The blue and red columns are not this row's category, so both receive 0.
- 4Write the row
In the order blue, green, red, the row reads 0, 1, 0.
The row is [0, 1, 0]. Exactly one column is 1, and it is the green column.
Running OneHotEncoder in Code
The source applies the encoder in two calls. First an encoder object is created. Then that object is fitted to the categorical feature and transforms it in one step, so fit_transform is the call that does both work. The result is converted to a dense array with toarray(), so it can be inspected and passed to a model that expects one. The code below follows that pattern on a small generated feature.
[[0. 0. 1.]
[0. 1. 0.]
[1. 0. 0.]
[0. 1. 0.]]
[array(['blue', 'green', 'red'], dtype='<U5')]- The feature is a two-dimensional array with one column, because the encoder expects a column of categorical values.
fit_transformlearns the categories from the data and produces the encoded matrix in one step.toarray()converts the encoder's output into a dense array that can be printed and inspected.- categories_ shows the categories the encoder learned, in the order that matches the output columns.
Each printed row has exactly one 1. The first row is red, so its 1 sits in the third column. The second and fourth rows are green, so their 1 sits in the middle column. The third row is blue, so its 1 sits in the first column.
Reading the Encoder's Output Columns
To read the matrix, you need the mapping from column positions to category names. The encoder keeps that mapping in the categories it learned during fit. The table below lists the rows from the example above, with the columns labeled by the categories shown in categories_.
| Row | Original color | blue | green | red |
|---|---|---|---|---|
| 0 | red | 0 | 0 | 1 |
| 1 | green | 0 | 1 | 0 |
| 2 | blue | 1 | 0 | 0 |
| 3 | green | 0 | 1 | 0 |
Encoded output for the color feature, with columns in the order the encoder learned them
Read each row by finding its single 1. The column that holds that 1 names the row's category, and the zeros in the other columns say the row is not any of those other categories.
Where an Encoded Result Diverges
When an encoded model underperforms, the fault is usually located at one of a few specific points. Tracing where the result departs from expectation is more productive than changing the model. The flowchart below follows the fit-and-transform path, from creating the encoder to passing the encoded matrix to a ridge regression. The point of the final step is to show the effect of the encoding on predictions, while the model itself stays unchanged.
The most useful checkpoint is the column order. The columns follow the categories the encoder learned during fit, not necessarily the order you listed them in your head. Printing categories_ after fitting shows that order directly, so you can compare it with what you expected before you trust the matrix.
Common Mistakes With Encoded Columns
Common mistake Treating an integer-coded category as a quantity with equal steps
What it looks likeCoding red as 0, green as 1, and blue as 2, then assuming green lies halfway between red and blue in its effect on predictions.
Why it failsA linear model learns one coefficient shared by every code, so each step along the codes adds the same amount.
Do this insteadEncode the categories as indicator columns so the model learns one coefficient for each category.
Common mistake Assuming the column order matches the order you typed the categories
What it looks likeExpecting red to be the first column after encoding a list that begins with red.
Why it failsThe columns follow the categories the encoder learned during fit.
Do this insteadPrint the learned categories and read each column against that list.
Common mistake Reading a row as having several categories at once
What it looks likeSeeing a row of 0, 1, 1 and concluding the row belongs to two colors.
Why it failsEach row has exactly one 1 in one-hot encoding, marking its own category.
Do this insteadIf a row shows more than one 1, check the input data and the encoding step, because the pattern is not what one-hot encoding produces for one categorical value.
Common mistake Forgetting to call toarray() before inspecting the result, which leaves the output in a form that is not a dense array
Tracing an Encoding Yourself
A feature called size has the values small, medium, large, and medium, in that order. Before running any code, write the one-hot row for each of the four entries, using the columns in alphabetical order: large, medium, small. Then check your rows against the printed categories and output of the encoder.
Hints
- Each row needs exactly one 1, and it goes in the column that matches its own size.
- Count the columns first: there are three categories, so each row has three values.
- The second and fourth rows should be identical, because both entries are medium.
Key Takeaways
- Integer codes make a linear model use one shared coefficient, so the effect of categories is forced onto one straight line across the codes.
- One-hot encoding creates one indicator column per category, with a 1 in the row's own category column and 0 elsewhere.
- With indicator columns, the linear model learns one coefficient per category, which matched the bike rental pattern much more closely than the integer encoding did.
- OneHotEncoder is created, fitted and used to transform the feature with
fit_transform, andtoarray()converts the result to a dense array. When the output looks unexpected, check the column order against the learned categories.