Clustering Data with Multiple Features

Most clustering methods hand you a finished set of groups. You get the final clusters and little else. Hierarchical clustering works differently. It shows you the whole history of how the groups came together, so you can see what the data looks like at every stage on the way to the final answer, not only at the end.

This article follows that journey step by step. It starts with agglomerative clustering, the kind of clustering that produces a hierarchical clustering. It then shows how each intermediate stage is itself a valid clustering, and it ends with the tool that makes the whole history easy to read even when the data has many features.

The process begins with every data point in its own cluster containing just itself. From there, the process repeatedly combines clusters. Over many steps, the single-point clusters grow into bigger groups until they become the final clusters.

It helps to think of this as a journey rather than a single assignment. Every point travels from being a single-point cluster to belonging to some final cluster. No point is assigned once and forgotten. Each point's cluster changes as the process runs, and the final cluster it ends up in is the result of a whole sequence of earlier merges.

The key fact about the snapshots is that each intermediate step is itself a valid clustering of the data. Every step groups the same points, but it uses a different number of clusters than the step before it. A step with many small clusters is a fine-grained view. A step with fewer, larger clusters is a coarser view of the same data.

Read the table of snapshots carefully. The step with four clusters is the finest possible view of these four points. The step with two clusters is coarser, and the step with one cluster is the coarsest. Every one of these stages is a complete answer to the question of how the points are grouped, so none of them is a throwaway intermediate result.

It is sometimes useful to look at all of these possible clusterings together. One approach is to overlay them on a single picture, so you can see how each cluster breaks up into smaller clusters as you move back through the history. The overlay makes the structure of the hierarchy visible at a glance.

The overlay gives a very detailed view. It shows how clusters split into smaller ones, and it lets you compare stages side by side. It has a hard limit, though. The overlay relies on the data being two-dimensional, meaning each point can be placed on a plane by two values. Because of that, it cannot be used on datasets with more than two features.

The dendrogram answers the same kind of question as the overlay: how do the clusters break up into smaller clusters? The difference is that it presents this structure as a tree-shaped diagram rather than as points on a plane. Clusters that merge early appear close together in the diagram, and the branching shows which groups joined which others on the way up to the final cluster.

Read the tree from the bottom up to follow the merges in order. This matches the run traced earlier: A and B join first, C and D join next, and the two groups join at the top. Read it from the top down to see how the final cluster splits into smaller clusters.

A dendrogram is a tool for visualizing hierarchical clustering, and it can handle multidimensional datasets with more than two features. The tree does not need a horizontal and vertical axis for each feature. It records only which groups joined and in what order, so the same reading works whether the points have two features or many.