Visualizing High-Dimensional Data

Most introductory machine learning examples start from a question that has a known answer, such as whether an email is spam or how much a house will sell for. The answer is attached to each example in advance, and the algorithm learns to predict it. Unsupervised learning is different. The data arrives without labels. Nobody has attached a correct answer to each example, so the algorithm cannot be told what to predict. Instead, it works directly with the data it is given.

This chapter looks at two kinds of unsupervised learning: transformations of the dataset and clustering. This article focuses on the first kind. Keep one question in mind as you read: if the data has no labels to guide the work, what can an algorithm usefully do with it? The answer this article develops is that it can describe the same data in a different way.

A dataset is always stored in some representation. Each example is described by a set of features, and the original representation is simply the set of features that were recorded. Think of a small table where every row is one example and every column is one feature. That table is one way of describing the data. Nothing forces the description to stay exactly as it is.

Trace the table below as you read. It is an illustrative example invented for this article, not data from the source. It has three examples, and each example is described by four recorded features. This is the original representation.

Two phrases in this definition carry most of the meaning. The first is new representation. The output is a different description of the same data. It is not a prediction and it is not a label. The second is easier to understand. The reason for creating the new description is usefulness. A transformation is judged by whether the new form helps a person or a downstream algorithm, not by whether it matches an answer that was given in advance.

The pipeline has no label input because none exists. The algorithm takes the features as they are and produces the new description directly. Labels therefore do not enter the process at any stage. This is what separates the transformation from a supervised method, where the known answers guide the learning.

A common application of unsupervised transformations is dimensionality reduction. Its starting point is a high-dimensional representation of the data, meaning a representation with many features for each example. Dimensionality reduction finds a new way to represent that same data so that the essential characteristics are still captured, but with fewer features.

Reduction to two dimensions is a common choice for visualization. Two features can be drawn on a flat page or screen, with one feature on each axis. A dataset reduced to two dimensions can therefore be plotted and looked at directly. A dataset with four or forty features cannot be drawn that way, because a flat plot has only two axes to carry them.

Each sample in the two-dimensional output receives one pair of coordinates, an x value and a y value. The pair is the sample's location on the plot. The values below are invented for illustration and are not from the source.

The same three examples now appear twice: once in their original representation and once in a new representation with two features each. The examples did not change identity. Only the description did. The diagram below compares the two states. The values are illustrative and generated for this article.