Understanding Principal Component Analysis

Principal Component Analysis, usually called PCA, is a way of re-describing a dataset. Instead of looking at the data through its original features, PCA rotates the data so that it is described by new features that are statistically uncorrelated with each other. It is often followed by a second step: keeping only the most important of those new features and discarding the rest.

The source material for this topic starts with a demonstration rather than with the theory. A plotting helper from the mglearn library generates a synthetic two-dimensional dataset, applies PCA to it, and shows the result in several panels. The first panel shows the original points. Later panels show the same points after PCA has rotated them, and after a reduction step has dropped one direction. As you read, your task is to work out what changes between panels and why those changes happen.

Before reading further, form a guess. If two features in a dataset tend to rise and fall together, what would you expect a rotation of the coordinate axes to do to that relationship? Hold your guess in mind while the next section traces the rotation step by step.

Start with the state of the data before PCA touches it. In the generated example the two features are correlated by construction, because the second feature is built from the first. When you plot the points, they form a tilted cloud: knowing where a point sits along one feature tells you roughly where it sits along the other. The off-diagonal entries of the matrix that summarizes the two features are therefore clearly nonzero, which is the numerical signature of that shared movement.

The rotation changes the axes, not the points. Before the rotation is applied, the points form a tilted cloud relative to the original feature axes. As the axes line up with the principal components, the cloud settles so that it sits along the new axes, and the correlation between the two new features disappears. This is the moment to look for in a trace of the code: before it, the cloud is tilted; after it, the cloud lies along the axes.

PCA does not start by looking at the original columns. It starts by looking for directions in the data. The first direction it looks for is the direction of maximum variance: the direction, or vector, that carries the most information. Put another way, it is the direction along which the features are most correlated with each other. The algorithm labels this first direction Component 1.

The first component is chosen first because it captures the most information. The procedure then searches for the direction that carries the most remaining information while being orthogonal, meaning at a right angle, to the first one. In two dimensions only one perpendicular orientation exists, so the second component is forced into that single position. In higher-dimensional spaces many orthogonal directions exist, and the procedure chooses among them the one that carries the most of what is left.

The right angle is what keeps the new features uncorrelated. If the second direction were allowed to lean toward the first, the two new features would still move together to some degree. Forcing the directions to meet at a right angle is what removes that shared movement.

A component is drawn as an arrow, but where its head and tail sit does not matter. The same line can be drawn from the center outward in either sense. What matters is the line's direction and the amount of variance it captures.

PCA produces as many principal components as the original features. Each one is a direction in the feature space, not one of the original columns of the table.

Before the rotation is applied, the mean is subtracted from the data. This moves the cloud of points so that it is centered around zero. The rotated representation that PCA produces is therefore centered around zero as well. Centering is a separate step from rotation, and it is easy to skip when you write the procedure yourself. The problem usually appears later, in the output, rather than at the point where the step was skipped.

PCA can also reduce the number of dimensions. After the components are found, you can retain only some of them. In the illustration the natural choice is to keep only the first principal component. That turns a two-dimensional dataset into a one-dimensional dataset.

The transformation is sometimes used to remove noise effects from the data, or to visualize which part of the information the retained components capture. The source material does not measure how much spread is lost in a specific case, so the amount lost depends on the particular dataset and should be checked rather than assumed.

A reduced PCA result can look clean and still be wrong. The safest way to find the problem is to walk through the pipeline in order and find the first stage where the data stops matching what the source describes. The diagram below lays out those checkpoints.