Principal Components and Variance

Imagine a dataset where each data point is described by two features, and the two features tend to rise and fall together. Knowing one of them tells you a lot about the other. That shared movement makes the data harder to describe compactly, because the two features are not independent pieces of information. Principal component analysis, usually shortened to PCA, is a method built to deal with exactly this situation.

The core idea is that PCA does not start by throwing features away. It first rotates the dataset into a new arrangement of axes in which the rotated features are statistically uncorrelated with each other. Only after that rotation does it often select a subset of the new features, ranked by how important they are for explaining the data. The two-part order matters: rotate first, then select.

Picture the points as a cloud drawn on the original two axes. Because the two features rise and fall together, the cloud is stretched along a tilted direction rather than sitting evenly inside the square the axes form. Each axis on its own only partly describes that tilt, so the axes are correlated.

Once the principal components are found, the data can be re-expressed so that the first principal component lines up with the horizontal axis and the second lines up with the vertical axis. The points themselves do not move relative to each other. Only the coordinate system used to describe them changes.

Two things happen during this rotation. First, the data is centered around zero, because the mean was subtracted before rotating. Second, the two axes in the rotated representation are uncorrelated. In matrix language, the correlation matrix of the transformed data has zeros everywhere except along its diagonal. The diagonal entries describe each axis with itself, so the off-diagonal entries, which would measure a relationship between different axes, all vanish.

PCA's first step is to look for the direction in the data that contains the most information. In plain terms, this is the direction along which the features are most correlated with each other. It is the line through the cloud of points where the points spread out the most, and it captures the largest share of what varies in the data. This first direction is the first principal component.

After finding that first direction, PCA searches for a second direction that contains the most remaining information while being orthogonal to the first, meaning at a right angle to it. In two dimensions there is only one possible perpendicular direction, so the search is trivial. In spaces with more dimensions, many directions can be orthogonal to the ones already chosen, so the choice is more open. The source describes this as potentially infinitely many orthogonal directions in higher-dimensional spaces.

Two properties of principal components are worth holding onto. First, principal components are orthogonal to each other, so each new direction sits at a right angle to the others. Second, the number of principal components equals the number of original features. A dataset with five features yields five principal components, even though those components are new directions rather than the original features themselves.

PCA becomes a dimensionality reduction tool when you keep only some of the principal components. Suppose the data has two dimensions. You can keep the first principal component and discard the second. The result is a one-dimensional description of each point, which is a much smaller representation than the original two-dimensional one.

A reduced representation is one-dimensional, so it cannot be drawn directly in the original two-dimensional space. PCA allows you to map the reduced data back. You undo the rotation and add the mean back to the data. The resulting points are back in the original feature space, but they now carry only the information contained in the first principal component.

This round trip is useful for two purposes the source describes. It can remove noise effects from the data, because the discarded directions are removed. It can also help you visualize which part of the information is retained by the principal components you kept.

Principal components and original features play different roles. The table below contrasts them so you can keep track of which idea applies at each step of PCA.