Choosing Machine Learning Algorithms

Machine learning algorithms need data in a consistent numeric form, so categorical variables must be encoded first.

Programming

A Puzzle Before the Model Starts

Imagine you are about to train a model on a table that records the color of each item. The column holds the words red, green, and blue. Before the model can learn anything, every value in that column has to be turned into something it can work with. The question below is a small test of the idea you will study in this article.

What do you think happens?

The categories are listed in the fixed order red, green, blue. Which vector represents the value green?

[1, 0, 0]
[0, 1, 0]
[0, 0, 1]
[1, 1, 1]

Reveal answer

Answer:

[0, 1, 0]

Green is the second item in the agreed list, so the slot at position 1 is set to 1 and every other slot is 0. The rest of this article explains why that works and why the list order has to stay fixed.

Why Models Need Numeric Input

Before a machine learning algorithm can learn anything, the data has to be written in a form the algorithm can work with. A model does not read a spreadsheet the way a person does. It works with numbers arranged in a consistent layout, so the same kind of information must always appear in the same place and in the same form.

This creates two practical questions. The first is what to do with data that is not numeric to begin with, such as a color, a city, or a product type. The second is, once the data is numeric, whether you can improve it by building new features out of the old ones. This article takes the first question first and then turns to the second.

The diagram below shows the path a categorical value takes. The raw label enters an encoding step, which compares it with the agreed list of categories and produces a numeric vector. Only the vector reaches the model.

How does text data such as red, green, and blue flow through an encoding step before it reaches the model, and what would break without that step?

Without this step, the model has no consistent way to compare one row with another. The text 'green' cannot be multiplied, added, or measured against 'blue' in any meaningful numeric sense. An encoding step is what makes a categorical value usable at all.

One-Hot Encoding as a Vector

One-hot encoding gives each category its own column. The column for the matching category holds a 1, and every other column holds a 0. A row for one value therefore contains exactly one 1, which is why the method is called one-hot. The agreed list of categories defines which position belongs to which category.

The method also avoids implying a false order. If you simply assigned red the number 1, green the number 2, and blue the number 3, the model could read green as lying between red and blue, or as twice red. One-hot encoding does not create any such ranking, because each category sits in its own slot with no numeric distance to the others.

Which position in the vector turns on for each category, and how does a single category like red become a row of 0s and 1s?

In the diagram, the value red matches the first slot in the agreed list, so that slot becomes 1. The other two slots stay at 0, giving the row [1, 0, 0]. Each category in the list gets its own column, so the number of columns equals the number of categories.

Tracing One-Hot Encoding in Python

The following Python code implements the idea directly with plain lists. The function creates a vector of zeros as long as the category list, finds the position of the value, and sets that position to 1.

Code walkthrough

colors = ['red', 'green', 'blue']

def one_hot(value, categories):
    vector = [0] * len(categories)
    position = categories.index(value)
    vector[position] = 1
    return vector

print(one_hot('red', colors))
print(one_hot('green', colors))
print(one_hot('blue', colors))

1
Line 1 defines the fixed category list. Its order is the contract that every row must follow.
2
Line 3 begins the function. It takes the value to encode and the list of categories.
3
Line 4 creates a vector of zeros whose length equals the number of categories, so for three colors it has three slots.
4
Line 5 asks the list where the value sits. For 'green' the answer is index 1, because Python counts from 0.
5
Line 6 sets that one slot to 1. Every other slot keeps its 0.
6
Lines 8 to 10 call the function for each color and print the resulting vector.

colors holds the agreed category list in a fixed order
[0] * len(categories) builds a row of zeros with one slot per category
categories.index(value) finds the position of the category in the list

Output

[1, 0, 0]
[0, 1, 0]
[0, 0, 1]

Worked example

Encoding a Value That Is Not First

Encode the value 'blue' using the category list ['red', 'green', 'blue'].

1
Find the position
'blue' is the third item in the list. Python counts from 0, so its index is 2.
2
Start from zeros
Create a vector with three slots, all set to 0: [0, 0, 0].
3
Set the matching slot
Change the slot at index 2 to 1, giving [0, 0, 1].
4
Check the row
Exactly one entry is 1, and it sits in the column belonging to blue.

Result

The encoded row for blue is [0, 0, 1].

Before

colors = ['red', 'green', 'blue']
print(colors.index('green'))

The code finds the position of green, which is 1, but the model still has no numeric row to read.

After

colors = ['red', 'green', 'blue']
vector = [0] * len(colors)
vector[colors.index('green')] = 1
print(vector)

The same position is used to switch on one slot, producing the row [0, 1, 0] that a model can consume.

Mistakes When Encoding Categories

Common mistake
Assigning category numbers such as red = 1, green = 2, blue = 3
What it looks like
```
encoded = {'red': 1, 'green': 2, 'blue': 3}
```
Why it fails
These numbers imply an order and distances that do not exist. The model may treat green as halfway between red and blue.
Do this instead
Use one-hot encoding so that each category has its own column with a 1 or a 0.
Common mistake
Changing the category order between rows or between training and prediction
What it looks like
Training uses ['red', 'green', 'blue'] but prediction uses ['blue', 'green', 'red']
Why it fails
The same color lands in different columns, so the model reads the same information as different information.
Do this instead
Fix the category list once and reuse it everywhere the data is encoded.
Common mistake
Expecting derived features to add information that was missing from the inputs
What it looks like
Believing that adding x squared gives the model facts it never had
Why it fails
Derived columns are rearrangements of the original inputs. What changes is the shape of the information the model receives.
Do this instead
Think of derived features as new routes to a pattern, not as new data.

Deriving New Features from Existing Columns

Feature engineering creates derived features from existing data. Expert knowledge can guide which ones to build. Three common kinds of derived feature are binning, polynomial terms, and interactions. Each one rebuilds information already present in the original columns into a different shape.

The diagram below shows how two original features, x1 and x2, can produce new columns. Squaring a feature and multiplying two features are both rearrangements of values the model already has.

How do the original features x1 and x2 combine into new columns such as x1 squared and x1 times x2, and how do these relate to the original inputs?

A linear model that can only weight x1 and x2 separately cannot express x1 times x2 on its own. Supplying that product as a column gives the model a new route to the pattern. The product itself was already implied by the inputs, but the model could not reach it without the extra column.

Matching Features to the Algorithm

The source for this article makes a broader claim: the features you use, and how well they match the method you choose, is often the most important factor in whether a machine learning approach works well. The choice of algorithm and the choice of features are not independent decisions.

The comparison concerns where the work goes. For a linear model, much of the effort of capturing nonlinear structure can be placed into the features, which is why binning and polynomial or interaction terms are useful. For random forests and SVMs, the source indicates that this explicit expansion may not always be necessary, because these methods can learn complex tasks without it. The practical question is, for each algorithm, how much of the required structure the features must supply and how much the method can supply itself.

Algorithm family	Where the structure usually comes from	Feature expansion
Linear model	Mostly from the features supplied	Binning, polynomial, and interaction terms are especially useful
Random forest	Can learn complex patterns from the method itself	The source indicates explicit expansion is not always necessary
SVM	Can learn complex patterns from the method itself	The source indicates explicit expansion is not always necessary

How the feature work differs by algorithm family, as described in the source.

Practice: Encode and Derive

EASY

Using the category list ['small', 'medium', 'large'], write the one-hot vector for 'medium'. Then, for a row with x1 = 4 and x2 = 5, list the squared and interaction columns that would be appended after the originals.

Hints

Count from 0. 'medium' is at index 1.
The vector should have three slots with exactly one 1.
x1 squared is 4 squared. x1 times x2 is 4 times 5.
The expanded row is the original values followed by the derived values.

Your attempt (not saved)

Key Takeaways

A model needs consistent numeric input, so categorical values such as colors must be encoded before the algorithm can use them.
One-hot encoding gives each category its own column, with a 1 in the matching column and 0 elsewhere, and it avoids implying a false order.
Binning, polynomial terms, and interactions are derived features built from existing columns. They reshape information rather than adding new facts.
The match between features and algorithm often matters as much as the algorithm itself, since linear models lean more on the features while random forests and SVMs may learn complex tasks without explicit expansion.

Related concepts

More concepts starting with C