Generating Synthetic Clustering Datasets with make_blobs and make_moons

A common first impression of k-means is that it is a general-purpose grouping tool. You choose a number of clusters, the algorithm runs, and groups come out. Many learners carry an unstated assumption with them: if the number of clusters is right, k-means will recover the groups that are really present in the data. This article tests that assumption. The short answer is that k-means can miss the groups even when the number of clusters is correct, because the shape of its answer is fixed by how it measures distance to centers.

To test the idea safely, we use synthetic datasets. Synthetic data is generated on purpose, so you know the true grouping in advance. That lets you compare what k-means predicts against a structure you already understand. The two generators in this topic are make_blobs, which produces round or stretched groups, and make_moons, which produces two interleaved half-circles. Together they cover the failure situations this article examines.

Suppose you generate a dataset with three real groups and set k to 3. It is tempting to think the algorithm must then find those three groups. The reasoning feels sound: the count matches, so the labels should match. But the count only controls how many regions the algorithm creates. It does not control the shape of those regions. If the true groups have shapes that the algorithm cannot draw, the correct count still produces a wrong grouping.

k-means defines each cluster solely by its center. Points are assigned by their distance to the nearest center. Because of this, the region belonging to each cluster is convex: a cluster can only be a shape you could fill by drawing straight lines between any two of its points. The algorithm can therefore capture only relatively simple shapes.

The second assumption concerns size. k-means treats all clusters as having a similar diameter. It places the boundary between two clusters exactly in the middle between their centers. Written as a rule, the boundary between centers $c_A$ and $c_B$ is the set of points where the distance to each center is equal, which is the midpoint $\frac{c_A + c_B}{2}$ together with the line perpendicular to the segment joining the two centers. Every point on one side of that line is closer to one center, so it receives that center's label.

Because the boundary is always the midpoint, one rule explains many surprising results. The boundary does not look at how many points a cluster has or how spread out its points are. It only looks at where the centers are. A large, spread-out cluster and a tight cluster are treated identically, provided their centers are in the same places.

The first failure case involves clusters that differ in density. Imagine a dense region in the lower left, a dense region in the upper right, and a sparser region in the center. A human looking at this picture would likely name these three groups: the dense lower-left region, the dense upper-right region, and the less dense central region. The expected grouping has three clear parts.

The data for this case uses three blobs with different standard deviations, so some are tight and some are loose. The loose center group has points that drift outward toward its neighbors. When k-means runs, the midpoint rule places each boundary halfway between centers. The sparse group's outer points can end up closer to a neighboring center than to their own group's center, so they are pulled across the boundary.

The source reports this outcome directly: cluster 0 and cluster 1 both contain some points that are far from the other points in their own cluster and reach toward the center of the plot. The algorithm has pulled the edges of the dense regions into neighboring clusters. The mismatch is not caused by a wrong k. It is caused by the boundary landing in the wrong place for groups of unequal spread.

The second failure involves clusters that are not round. k-means assumes that all directions are equally important for each cluster. Picture a dataset with three clearly separated parts, where each part is stretched along a diagonal rather than being a circle. The groups remain easy to see, but the stretching means that a point along the long axis of one group can be closer to the center of a neighboring group than to its own group's center.

The source generates this kind of data by creating a blob dataset and then applying a random linear transformation to the coordinates. That matrix multiplication stretches the round blobs into diagonal ellipses. The transformation changes the shape of each group but leaves the group membership of every point unchanged. The true grouping is still the same; only the geometry has changed.

The source states the reason plainly: k-means only considers the distance to the nearest cluster center, and it cannot handle this kind of data. A stretched group has no single center that represents all of its directions equally. The algorithm sees one center and one distance, so it cannot represent a group that extends far in one direction and little in another.

The third failure involves clusters with complex shapes. The source uses the two-moons dataset from Chapter 2, where two interleaved half-circles form two groups. Visually, the correct answer is obvious: each half-moon is one cluster. The two arcs curve around each other, so no straight midpoint line can separate one arc from the other without crossing both.

The source is direct about the outcome: here we would hope that the clustering algorithm can discover the two half-moon shapes, but this is not possible using the k-means algorithm. Because each cluster is convex and separated by a straight midpoint boundary, k-means cuts across the moons instead of separating them. The number of clusters is correct, so the failure has nothing to do with choosing k.

When a clustering result looks wrong, the useful question is where it departs from what you expected. A reliable way to find that spot is to plot predicted labels against the structure you believe is in the data, then check each region in turn. Start with the region that looks most mismatched. Ask whether the points there are close to their assigned center, and whether a neighboring center is nearly as close. In the varying-density case, the mismatch sits along the edges of the dense regions, where far-reaching points get pulled into a neighbor.

Once you have located the divergent points, the pattern tells you which assumption failed. Mismatches along the edges of dense regions point to the equal-size assumption. Mismatches that extend along a diagonal stretch point to the spherical assumption. Mismatches that cut across a curved arc point to the convex-shape limit. Reading the pattern this way turns a wrong answer into information about the data's geometry.