Principal Component Analysis for Dimensionality Reduction

This article works in the opposite direction from most explanations of t-SNE. Instead of starting with the mathematics, you start with a small Python program that reduces a dataset of handwritten digits to two dimensions, and then you use the program's output to ask what the result means. Each step of the program is a question about the data: what does the input look like, what does a simpler method produce from the same input, and what does t-SNE produce that differs from it?

The habit you will build here matters more than any single setting: keep track of where each move in the program happens. When a result surprises you, the location of the surprise tells you which step to examine. The program runs in three moves. First you load the images, then you reduce them with PCA as a baseline, and then you reduce them with t-SNE.

The digits dataset is included in scikit-learn, so you load it with a single function call. Every data point in it is an 8 by 8 grayscale image of a handwritten digit between 0 and 9. The dataset also carries a label for each image that tells you which digit it shows. Those labels are used only for coloring and checking the plots, because t-SNE itself never reads them.

Think of each data point as a picture first and a row of numbers second. The picture is what you look at when you draw a digit. The row is what a reduction method receives. Each image contributes 64 pixel values, and those values are what the methods work with. When PCA or t-SNE finishes, that same image has become a point with just two coordinates.

Before trying the new method, the program builds a baseline with PCA, which is a linear dimensionality reduction technique. The PCA model is fitted on the digits data, and then the same data is transformed onto the first two principal components. Each image, which has 64 pixel values, becomes a point with two coordinates. Because the two output axes are principal components, you can keep working with the fitted model and use those axes again.

The plot is a scatter in which the labels themselves reveal the structure. Reading the source's interpretation: zero, six, and four are relatively well separated along the first two components, although they still overlap, and most of the other digits overlap significantly. Keep this observation in mind, because it is the baseline you will compare against.

The most visible difference between the two methods is the sequence of calls. PCA is used in two steps, a fit followed by a transform. The TSNE class does not support transforming new data, so you use fit_transform, which builds the model and immediately returns the transformed data in one step. The flowchart below traces both sequences.

Now apply t-SNE to the same dataset so the two results can be compared. The source describes the t-SNE output as remarkable. All the classes are quite clearly separated. The ones and nines are somewhat split up, but most classes form a single dense group. t-SNE is unsupervised: it has no knowledge of the class labels. It still finds a two-dimensional representation that clearly separates the classes, using only how close points are to one another in the original space. The labels were used only afterward, to color and identify the points on the plot.

Put the two results next to each other and the differences become a set of questions you can answer from the code and the source text. PCA is a linear technique, so its output axes are principal components you can keep using with the fitted model. t-SNE is a non-linear technique, and its output comes from a single fit_transform call on the data you supplied. The diagram and table below summarize these differences.

The t-SNE algorithm has tuning parameters, and two of them are perplexity and early_exaggeration. The source explains that the algorithm often works well with its default settings, so the program does not change either one. The source invites you to experiment with both, but it also warns that their effects are usually minor.

This is useful to know before you start. If you change perplexity or early_exaggeration and see only a small shift in the picture, that is consistent with what the source describes. A large change would be unexpected from these settings alone. In that case, check the rest of the code first, such as the input passed to fit_transform, before blaming the tuning parameters.