Feature Engineering Process

An interaction feature is formed by multiplying two existing columns.

Programming

Why a Model Needs Numbers in a Fixed Layout

Before a machine learning algorithm can learn anything, the data has to be written in a form the algorithm can work with. A model does not read a spreadsheet the way a person does. It works with numbers arranged in a consistent layout, so the same kind of information must always appear in the same place and in the same form.

This raises two questions. First, what do you do with data that is not numeric to begin with, such as a color, a city, or a product type? Second, once the data is numeric, can you improve it by building new features out of the old ones? This article takes both questions in turn, and then connects them to the choice of algorithm.

What do you think happens?

Suppose a column holds the colors red, green, and blue. Before reading on, what do you think a learning algorithm would need in order to use that column?

Reveal answer

Answer:

It needs the colors converted into numbers in a consistent layout, because the algorithm works with numeric values arranged in a fixed structure.

A model cannot take the word red as input directly. The categories have to be encoded first, so every row carries the same kind of numeric information in the same place.

Why Categorical Variables Must Be Encoded

A categorical variable takes one of a limited set of labels rather than a number on a scale. Colors, cities, and product types are examples. Machine learning algorithms need data in a consistent numeric form, so these labels must be encoded before the algorithm sees them.

How does the raw value red move through the pipeline into numeric input for the algorithm?

Read the pipeline from left to right. The raw label is matched against an agreed list of categories, and the position of that match decides which output slot becomes 1. The algorithm never receives the word red. It receives only numbers in a layout that stays the same for every row. Without this step, the algorithm has no numeric values to process, so the learning step cannot begin.

Tracing One-Hot Encoding Row by Row

One-hot encoding gives each category its own column. The column for the row's category holds a 1, and every other column holds a 0. The vector therefore has as many positions as there are categories. With three colors, each row becomes a vector of length three.

Which position receives the 1 when the category is red, and how does the vector length relate to the number of categories?

The diagram shows one row. Each slot in the vector corresponds to one category, and the matching slot is the only one set to 1. The 1 sits in the first position because red is first in the agreed category list. If the row had been green, the 1 would move to the second position and the first position would hold 0.

The same logic can be written in plain Python. The function below builds a list of zeros, finds the position of the value in the category list, and sets that position to 1.

Code walkthrough

categories = ["red", "green", "blue"]

def one_hot(value, categories):
    vector = [0] * len(categories)
    position = categories.index(value)
    vector[position] = 1
    return vector

rows = ["red", "blue", "green", "blue"]
for row in rows:
    print(row, "->", one_hot(row, categories))

1
Line 1 defines the agreed category list, whose order determines the column positions.
2
Lines 3 to 7 define the function: create zeros, locate the category, set its slot to 1, and return the vector.
3
Line 9 defines a sample column of four rows.
4
Lines 10 and 11 apply the function to each row and print the result.
5
The output shows that blue appears twice and produces the same vector both times, which is the consistency the algorithm needs.

The category list fixes the order of the columns once, and every row uses the same list.
The vector starts as all zeros, with one slot per category.
index() finds the position of the value in the agreed list.
Setting that single position to 1 produces the one-hot pattern.

Output

red -> [1, 0, 0]
blue -> [0, 0, 1]
green -> [0, 1, 0]
blue -> [0, 0, 1]

Worked example

Encoding a value that is not first in the list

Using the category list red, green, blue, what is the one-hot vector for the value blue?

1
Start with zeros
There are three categories, so the vector begins as three zeros: 0, 0, 0.
2
Locate the category
blue is the third item in the list, so its position is the third slot.
3
Set the matching slot
Change the third slot from 0 to 1, leaving the others at 0.

Result

The one-hot vector for blue is 0, 0, 1.

Why One-Hot Encoding Avoids a False Order

One-hot encoding avoids implying a false order. If the colors were simply numbered 1, 2, and 3, an algorithm could read blue as larger than red or as lying between the other two. The categories have no such ranking, so giving them numeric codes in a single column would introduce a relationship that does not exist. Giving each category its own column means no category is numerically closer to another.

Common mistake
Numbering categories in one column and treating the numbers as if they were measurements
What it looks like
Coding red as 1, green as 2, and blue as 3 in a single column.
Why it fails
This suggests blue is three times red and that green sits between them, which is a false order.
Do this instead
Give each category its own column, with a 1 in the matching column and 0 elsewhere.
Common mistake
Rebuilding the category list differently for each batch of rows
What it looks like
Using the list red, green, blue for one batch and blue, red, green for the next.
Why it fails
The same color now lands in a different column, so the layout is no longer consistent.
Do this instead
Fix the category list once and reuse it for every row.
Common mistake
Passing raw text such as red directly to an algorithm that expects numbers

Deriving New Features From Existing Columns

Feature engineering creates derived features from existing data, and expert knowledge can guide which ones to build. The three techniques below are binning, polynomial terms, and interaction features. Each one starts from columns the dataset already contains.

Matching Features to the Algorithm

The source makes a broader claim that frames everything above. The features you use, and how well they match the method you choose, is often the most important factor in whether a machine learning approach works well. The same feature set can perform differently depending on the algorithm it is paired with.

The comparison is about where the work goes. For a linear model, much of the effort of capturing nonlinear structure can be placed into the features, which is why binning and polynomial or interaction terms are useful. For random forests and SVMs, the source indicates that this explicit expansion is not always necessary, because these methods may learn complex patterns without it.

Algorithm	How much structure the features must supply	How much the method can supply itself
Linear model	A large share, so binning, polynomial terms, and interactions are often needed	Limited for nonlinear patterns on its own
Random forest	Explicit expansion is not always necessary	Can learn complex tasks without explicit expansion
SVM	Explicit expansion is not always necessary	Can learn complex tasks without explicit expansion

Where the work goes for each algorithm, as described in the source

Practice: Encode and Derive

EASY

Using the category list ["red", "green", "blue"], write the one-hot vector for the value green. Then state what an interaction feature would be for two columns with values 3 and 4.

Hints

Count the position of green in the list, starting from position 1 for the first item.
Only one slot in the vector should be 1; the rest should be 0.
An interaction feature is the product of the two original values.

Your attempt (not saved)

MEDIUM

A column holds the values small, medium, and large, and you want to feed it to a linear model. Explain why you should not number them 1, 2, and 3 in one column, and describe what you would do instead.