Data Representation for Machine Learning

Imagine you are preparing a table of information about customers so that a machine learning algorithm can learn from it. Some columns hold numbers, such as an income or an age. Others hold labels, such as a city name or a product color. A learning algorithm cannot simply read a label the way a person does. The data has to be represented in a form the algorithm can work with, and the way you represent it can change how well the final model performs.

This article follows one idea: feature engineering is the work of deciding how the inputs to a model should look. Instead of accepting every column exactly as it arrived, you create new features from existing data. Some of those features make a category usable, some reshape a number into groups, and some combine several numbers into new terms. As you read, keep one question in mind: what does the model need the data to look like in order to learn from it?

The same real-world information can be written down in many different ways, and each way changes what the system can make of it. Choices about representation are made before learning begins, and those choices shape what the system can learn. The step sits at the very start of the learning process, so it is worth understanding the question before looking at any particular method.

The idea names two linked questions. The first is how the data should be represented: how it is expressed so that machine learning can work with it. The second is which data aspects to focus on: which parts of the data deserve attention. Both questions are needed. A representation that captures the wrong aspects of the data will not help the learner, and a well-chosen set of aspects still needs a usable form.

Feature engineering means engineering new features. A new feature is often derived from one or more features that already exist in your data. The person creating the feature can use expert knowledge, meaning knowledge about the problem domain, to decide which derived features are likely to be useful. Someone who understands the problem may know that a certain combination of measurements matters more than any single measurement alone.

The techniques in this article are four common ways to create derived features: one-hot encoding for categorical variables, binning for continuous variables, polynomial terms, and interaction terms. One-hot encoding is a way of representing data. The other three are ways of expanding the set of features the model sees.

A categorical variable takes one of a limited set of labels, such as red, green, or blue. One-hot encoding replaces one column of labels with several columns of numbers, one column for each possible category. For any given row, the column that matches the row's category holds a 1, and every other category column holds a 0.

The following example is an original illustration written for this article. It uses a plain Python list of colors and builds one row of 0s and 1s per entry. The categories are sorted so the column order is predictable: blue comes first, then green, then red.

Trace the first row through the loop. The label is red. Blue does not match, so its column holds 0. Green does not match, so it holds 0. Red matches, so its column holds 1. The result is [0, 0, 1]. The fourth entry is also red, so it produces the same row. The representation is a table that a learning algorithm can read, even though it no longer contains the word red.

Notice that each 1 sits in exactly one position per row. The number of columns equals the number of possible categories. Because each category now has its own column, the algorithm can treat red, green, and blue as separate inputs rather than as an ordered scale.

A continuous variable, such as an age or a temperature, can take any value within a range. Binning creates a new feature by grouping continuous values into bins. Each bin covers a range of values, and a value is assigned to the bin whose range contains it. The idea is to replace a fine-grained number with the index of the group it falls into.

The example below is an original illustration. The bin edges define four ranges: below 18, from 18 up to 35, from 35 up to 65, and 65 and above. The function returns the index of the bin that each age falls into. The edges are a choice made for this example, not a rule from the source.

Polynomial terms are new columns built from an existing numeric feature by raising it to a power. The simplest example adds a column for $x^{2}$ next to the original column $x$. The source describes adding polynomials as one of the ways to create new features for linear models.

Why does the new column help? A model that combines the original $x$ column in a straight-line way can only draw a straight relationship between $x$ and the output. Once a column holding $x^{2}$ is available, the model can combine both columns, so it can represent a curve. The model is still linear in its inputs, but the inputs now include a curved version of the original feature. Adding the column changes what the model is able to express, not the data itself.

An interaction term is a new column formed as the product of two existing features. The source describes interactions as one of the ways to build new columns for linear models, alongside binning and polynomials. The new column is useful when the effect of one feature depends on the value of another.

The example below is an original illustration using a small sales table. Each sale has a unit price and a number of units bought. Multiplying the two gives a column that tracks total spend for each sale.

Look at sales A and C, which both bought ten units, and sales A and B, which share a unit price of 2. Price alone does not separate the large purchases from the small ones, and units alone ignore how much each unit costs. The interaction column shows the combined effect directly: 20 for A, 2 for B, 90 for C, and 9 for D. Neither feature by itself carries that combined information, but the product does.

The source draws a contrast between two families of models. Linear models may benefit greatly from new features created by binning, polynomials, and interactions. More complex nonlinear models, such as random forests and SVMs, might learn complicated tasks without the feature space being expanded explicitly. The choice of technique therefore depends on the kind of model you are training.

Consider a small, invented case. A shop keeps a record for each sale. Every record includes a date, a product name, a price, and the name of the store branch. A team wants a machine learning system to help predict how much a customer will spend. The worked example below applies both questions to this record.