SciPy Clustering Functions

Hierarchical clustering is often easier to understand from a picture than from a definition, so this article starts with a short code path.

Programming

Predicting What ward() Returns

Hierarchical clustering is often easier to understand from a picture than from a definition, so this article starts with a short code path. The workflow has three stages: a SciPy function computes a linkage array from the data, a second SciPy function draws that linkage array as a dendrogram, and then a horizontal cut through the drawing picks out clusters. Your job at this stage is to notice which object each call produces and which object the next call consumes.

What do you think happens?

You call linkage_array = ward(X) on a 2D data array X, and then pass linkage_array to dendrogram. Before reading further, what do you think linkage_array holds?

A final cluster label for each point, such as 0, 1, 2
A list of merges, recording which clusters were joined and how far apart they were
A plotted image of the dendrogram
The chosen number of clusters

Reveal answer

Answer:

A list of merges, recording which clusters were joined and how far apart they were

The function takes the data array and computes a linkage array that encodes the hierarchical similarities between clusters. Because it is a list of merges and their distances, the whole tree can be reconstructed from it. Its name and its role as the input to dendrogram both point to a merge history rather than a set of labels.

Notice that the calls never ask for a number of clusters while the tree is being built. The count only enters when you draw a horizontal cut across the finished tree. That detail matters for the rest of this article, because it means the tree itself carries more information than any single set of labels.

Why SciPy Returns a Merge History

A scikit-learn clustering object is usually fitted and then exposes labels, one per data point. The SciPy function works differently: it takes the data array and computes a linkage array, which encodes the hierarchical similarities between clusters. The output is therefore a history of the merging process rather than a final assignment of points to groups.

Aspect	scikit-learn estimator	SciPy clustering function
Typical workflow	Create the object, fit it, then read its labels	Pass the data array to the function to get a linkage array
Main output	A final label for each point	A merge history of clusters and distances
What you can do next	Use the labels directly	Draw a dendrogram, then choose a cut to obtain labels at a chosen level
Where the cluster count appears	Usually set when the estimator is created	Not needed to build the tree; it appears when a horizontal cut is drawn

Comparing the outputs of the two clustering styles

Definition

A dendrogram is a tree-shaped plot of hierarchical clustering that works for datasets with more than two features. Because it is drawn from the merge history rather than from a scatter of the raw points, it can represent data that a simple two-dimensional scatter plot cannot show.

The tree is built from pairwise distances between clusters, not from the picture of the points. Each row of the linkage array describes one merge, so the complete sequence of merges is enough to rebuild the entire tree and draw it.

python

import numpy as np
import matplotlib.pyplot as plt
from scipy.cluster.hierarchy import ward, dendrogram, fcluster

X = np.random.rand(10, 2)

# Stage 1: compute the merge history
linkage_array = ward(X)

# Stage 2: draw the merge history as a tree
dendrogram(linkage_array)
plt.show()

# Stage 3: choose a horizontal cut and read the resulting labels
labels = fcluster(linkage_array, t=2, criterion="maxclust")
print(labels)

import numpy as np
import matplotlib.pyplot as plt
from scipy.cluster.hierarchy import ward, dendrogram, fcluster

X = np.random.rand(10, 2)

# Stage 1: compute the merge history
linkage_array = ward(X)

# Stage 2: draw the merge history as a tree
dendrogram(linkage_array)
plt.show()

# Stage 3: choose a horizontal cut and read the resulting labels
labels = fcluster(linkage_array, t=2, criterion="maxclust")
print(labels)

Notes on the code

ward(X) consumes the data array and returns the linkage array, which is a merge history
dendrogram(linkage_array) consumes that same linkage array and draws the tree
fcluster is used here to choose a cut that produces two clusters; the tree itself never required a cluster count

Reading a Dendrogram from the Bottom Up

Once the plot is drawn, the structure is read in a fixed direction: from the bottom upward. The data points sit along the bottom and are numbered. Each point starts as its own single-point cluster, and these are the leaves of the tree. Whenever two clusters are joined, the diagram adds a new parent node that connects them. Reading from the bottom therefore replays the algorithm in the order the merges happened.

Starting from the individual points at the bottom, which two items joined first, and what larger clusters form as you move upward? The first join is points 1 and 4, then points 6 and 9, and the top level splits into two branches.

Worked example

Tracing the source example from the bottom up

Using the merge order described for the source example, determine which points were joined first and what the two largest clusters are.

1
Find the first join
The first join is between points 1 and 4, which the source identifies as the closest pair.
2
Find the second join
The next join combines points 6 and 9. The tree keeps growing upward from there.
3
Identify the top level
Continuing upward, the tree reaches a top level with two branches. One branch holds points 11, 0, 5, 10, 7, 6, and 9. The other holds points 1, 4, 3, 2, and 8.
4
Name the two largest clusters
These two branches are the two largest clusters visible on the left side of the plot.

Result

The first merges are 1 with 4 and then 6 with 9. The two largest clusters are the branch with points 11, 0, 5, 10, 7, 6, 9 and the branch with points 1, 4, 3, 2, 8.

Key point

Reading upward replays the merges in order: the lower a join sits, the earlier it happened, and the higher a join sits, the larger the clusters it combines.

Branch Heights and Choosing a Cut

A dendrogram does more than show order. Its vertical axis gives the distance at which clusters merge, and the length of each branch shows how far apart the clusters being joined actually were. A short branch means the merge combined nearby clusters. A long branch means the algorithm had to bridge a large gap to combine them.

A horizontal cut across the tree defines a set of clusters. Every branch the cut crosses becomes one cluster. Raising the cut merges more branches together, so fewer clusters remain, and lowering it splits them apart, so more clusters appear. The cut is therefore the place where you decide how many clusters you want.

Across the steps from raw data to ward, linkage, dendrogram, and the chosen cut, which step is where a wrong cluster count first appears? The count is set only at the cut, after the tree is already built.

The source example makes this concrete. The longest branches are the ones marked by the dashed line labeled three clusters. Those long branches show that moving from three clusters to two required merging some very far-apart points. The same pattern appears at the top of the chart, where combining the last two clusters into one bridges a relatively large distance. Long branches near a cut are a signal that the clusters on either side are genuinely separated.

The source draws two such cuts, one at height 7.25 and one at height 4. Comparing where the two cuts cross the branches shows how the same tree yields different groupings depending on the height you choose.

Mistakes That Produce Surprising Plots or Counts

Common mistake
Expecting the tree to already contain a cluster count
What it looks like
Looking at the dendrogram and assuming the number of clusters is fixed by the plot itself.
Why it fails
The code never asks for a cluster count while building the tree. The count appears only when a horizontal cut is drawn.
Do this instead
Decide the count at the cut, and change the cut height to explore other counts without rebuilding the tree.
Common mistake
Reading the dendrogram from the top down
What it looks like
Starting at the largest branch and tracing downward to explain which points merged first.
Why it fails
Each point starts as its own leaf at the bottom, and merges happen in the order you read upward.
Do this instead
Begin at the leaves and move upward to replay the merges in order.
Common mistake
Treating a long branch as a bad merge automatically
What it looks like
Dismissing a tall join as an error without checking what it bridges.
Why it fails
Branch length is the distance bridged by the merge. A long branch can be a genuine separation between groups.
Do this instead
Use long branches near a cut as evidence of separation, and check the distance before deciding anything is wrong.
Common mistake
A dendrogram reflects the merges the algorithm made, so a poor merge appears on the plot just as faithfully as a good one

Edge case

A dendrogram does not change what agglomerative clustering can separate. The source notes that agglomerative clustering still fails to separate complex shapes such as the two_moons dataset. The dendrogram will show those merges faithfully, including the poor ones. The source points to DBSCAN as the next algorithm that handles that kind of shape.

MEDIUM

A colleague says the dendrogram shows three clusters, but your fcluster call with t=2 and criterion maxclust returns two labels. Using the three-stage workflow, which stage is responsible for the count you see, and where would you look first?

Hints

Ask which stage consumes a cluster count and which stages only build or draw the tree.
The count is set when a horizontal cut is applied, so compare the cut you drew with the count you expected.

Your attempt (not saved)

Practice Reading a Merge Sequence

MEDIUM

Given a dendrogram where the first join is between two nearby points and the top level has two long branches, explain in your own words which merges happened first, what the top level represents, and where you would place a horizontal cut to obtain two clusters.