Preprocessing Data for Machine Learning

Most introductions to machine learning start with a model that makes a prediction and gets scored against a known answer. Unsupervised algorithms work differently. They look at data that has no answer attached and try to find structure in it. That raises a practical question for anyone working with data: if nothing is being predicted, what is the algorithm actually for?

The answer comes in two parts. In the first, the algorithm is a tool for a person who wants to understand a dataset. In the second, the algorithm is a preprocessing step that prepares data for a later supervised stage. This article follows both roles, and then turns to the preprocessing steps that make data usable at all: encoding categories and building new features from existing ones.

Unsupervised algorithms are often used to understand data in an exploratory setting. Here a data scientist wants to see better what a dataset contains before deciding what to do with it. The purpose is insight, not a deployed system. The algorithm is a tool for the person looking at the data, helping them notice structure they would otherwise miss.

In this setting the unsupervised algorithm is usually not part of a larger automatic system that runs on its own. Its output goes to a person who reads it, questions it, and decides on the next step. Keeping this in mind prevents a common confusion: an unsupervised result used for understanding is a different kind of thing from an unsupervised step wired into a production pipeline.

Preprocessing means transforming data before a later stage uses it. In the second application, the later stage is a supervised algorithm, one that learns from labeled examples. Instead of handing the supervised algorithm the raw data, the data first passes through an unsupervised step that learns a new representation of it.

Follow one dataset through both applications. In the exploratory role, a data scientist runs the unsupervised algorithm, looks at the structure it reveals, and forms questions. Nothing downstream consumes the result automatically. In the preprocessing role, the same kind of algorithm runs first, its new representation is passed on, and a supervised algorithm learns from that representation together with the known answers. The data moves through the same first stage in both cases, but the reason for running it and the consumer of its output are completely different.

The source describes this idea in general terms and does not name a specific unsupervised method, so the following code is a generated illustration of the mechanical shape of a representation change. Each original point has two measurements. Each point is rewritten as a new pair of values computed from those two measurements. Here the new pair is the sum and the product of the measurements, chosen only to make the transformation visible.

A new representation can bring two practical benefits to supervised learning. It can sometimes improve the accuracy of the supervised algorithm. It can also sometimes reduce memory and time consumption. Both are possibilities, not promises.

The hedged wording matters. A preprocessing step adds its own work, so the benefit has to outweigh the cost of running it. A learner should treat preprocessing as a hypothesis to test against the supervised result, not as a step that always helps. The only reliable check is to train the supervised algorithm with and without the step and compare the results.

Before a machine learning algorithm can learn anything, the data has to be written in a form the algorithm can work with. A model does not read a spreadsheet the way a person does. It works with numbers arranged in a consistent layout, so the same kind of information must always appear in the same place and in the same form.

One-hot encoding gives each category its own column. The column for the matching category holds a 1, and every other column holds a 0. Because each category gets its own slot, the encoding does not imply a false order between categories. Read the diagram below from left to right: the incoming label is matched against a fixed list of categories, and the position of that match decides which output slot becomes 1.

The list order must stay fixed across the whole dataset. If the order changed between rows, the same color would map to different columns in different rows, and the model would receive contradictory information.

Feature engineering creates derived features from existing data, and expert knowledge can guide which ones to build. Three common forms are binning, polynomial terms, and interactions. Binning splits a numeric column into ranges. A polynomial term adds a column that is a power of an existing column, such as its square. An interaction adds a column that is the product of two existing columns.

Notice that none of these new columns contain information that was not already in the original inputs. The bins, squares, and products are rearrangements of what was already there. What changes is the shape of the information the model receives.

The features you use, and how well they match the method you choose, is often the most important factor in whether a machine learning approach works well. For a linear model, much of the effort of capturing nonlinear structure can be placed into the features, which is why binning and polynomial or interaction terms are useful. For random forests and SVMs, the explicit expansion may not be necessary, because these methods may learn complex tasks without it.