Linear Transformations

Why Fewer Coordinates Can Help

Suppose each vector in a dataset has d features. Working with every feature can be unnecessary when a smaller representation can retain the important structure of the data. Dimensionality reduction addresses this by replacing each vector with a representation that has fewer dimensions. In PCA, this replacement is a linear transformation followed by a reconstruction process, so the reduced representation is judged by how closely the original vectors can be approximated.

The Compression Step

Start with one vector x in R^d. The compression matrix W has n rows and d columns, with n smaller than d. Multiplying W by x produces y = W x, a vector in R^n. Because n is smaller than d, y contains fewer coordinates than x. This move from R^d to R^n is the central dimensionality-reduction step.

The Recovery Step

Compression alone gives the smaller vector y. PCA also describes how to return to the original dimensional setting. The recovery matrix U has d rows and n columns. Multiplying U by y produces the recovered vector, written as x̃ = U y. This recovered vector lies in R^d, just as the original x does, but it is generally an approximation rather than a claim that every detail of x has been restored exactly.

What PCA Optimizes

The matrices W and U are not selected merely because their shapes are compatible. PCA looks for the pair that makes recovered vectors close to the original vectors across the dataset. If the dataset contains vectors x1 through xm, each vector is compressed and recovered as x̃i = U W xi. PCA chooses W and U to minimize the total squared distance between each original vector xi and its recovered version x̃i.

The objective connects both transformations. W controls the move into the smaller space, U maps the compact representation back into the original space, and the squared distance evaluates the combined result. A pair of matrices is useful in PCA only when this combined process keeps the recovered vectors close to the originals across the dataset.

Common Reasoning Mistakes

• Treating compression as if it preserves the original number of coordinates.

  The compression matrix W has n rows with n smaller than d, so y = W x belongs to R^n.

  Fix: Track the output dimension from the number of rows of W.

• Assuming recovery exactly restores the original vector.

  The source describes x̃ as an approximation and evaluates it by its distance from x.

  Fix: Use the phrase recovered approximation and compare it with the original using squared distance.

• Describing PCA as compression without an accuracy objective.

  PCA chooses W and U to minimize the total squared distance between original and recovered vectors.

  Fix: Explain both stages: dimensionality reduction and reconstruction-error minimization.

• Checking only whether matrix dimensions are compatible.

  Compatible shapes allow the operations, but PCA selects the matrices according to how closely the recovered vectors match the originals.

  Fix: Separate shape compatibility from the optimization goal.

Trace the Transformation

1. Dimensionality reduction maps an original vector from R^d into a smaller space R^n, where n is less than d. The compression matrix W has n rows and d columns and produces y = W x. The recovery matrix U has d rows and n columns and produces the approximation x̃ = U y in R^d. PCA chooses W and U by minimizing the total squared distance between original vectors and their recovered approximations.