Linear Subspaces
PCA's optimization problem is a reconstruction problem: choose U and W to minimize total squared distance.
From Compression to Recovery
PCA can be understood as a reconstruction problem. It searches for two matrices that work together: W compresses an original vector x, and U uses that compressed representation to produce a recovered vector. The recovered vector is not an unrelated replacement for x. It is specifically the result of applying the combined mapping UW to x, written as UWx.
Tracing the Mapping
The mapping has two stages. First, W changes x into a compressed representation Wx. Next, U uses Wx to produce UWx. Combining the two stages gives one mapping from the original vector directly to its recovered version: x ↦ UWx. This notation matters because the reconstruction comparison is between x and the output produced by this complete mapping.
Following one vector
Given an original vector x, describe the role of W, U, and UWx in the PCA reconstruction process.
Compression: W acts on x and produces Wx, the compressed representation of the original vector.
Recovery: U acts on Wx and produces UWx, the recovered vector.
Comparison: The recovered vector UWx is compared with x when PCA evaluates reconstruction quality.
The recovered vector is UWx because it is the output of applying W first and U second.
The Reconstruction Objective
PCA chooses U and W to minimize the total squared distance between original vectors and their recovered versions. For an original vector x, the recovered version used in that comparison is UWx. Across the data being reconstructed, PCA therefore prefers a matrix pair whose combined mapping produces recovered vectors close to the corresponding originals under the squared-distance objective.
The optimization does not minimize the distance between x and Wx. Wx is the compressed representation. The reconstruction comparison uses x and UWx, because UWx is the recovered vector in the original output space.
Structure of the Solution
The minimizing solution has two connected properties. First, the columns of U are orthonormal, expressed by UᵀU = I. Second, W is the transpose of U, expressed by W = Uᵀ. Thus W is not an unrelated matrix selected independently from U. The two matrices have a precise relationship, and their combined mapping is UW.
The Subspace of Recovered Outputs
Fix U and W, then let x vary over possible input vectors. Every resulting recovered vector has the form UWx. The collection of all such outputs is the range of the mapping UW. The source describes this range as an n-dimensional linear subspace of Rᵈ.
This geometric view changes how to interpret PCA. PCA is not merely producing separate recovered vectors one at a time. It selects a matrix structure whose possible recovered outputs occupy one shared n-dimensional linear subspace of Rᵈ. Each particular input x produces one point in that subspace through UWx.
Comparing two inputs
Suppose two different inputs, x₁ and x₂, are processed using the same fixed matrices U and W. Describe what can be said about UWx₁ and UWx₂.
Apply the same mapping: Each input is sent through the same combined mapping UW, producing UWx₁ and UWx₂.
Identify the collection: Both recovered vectors belong to the range of UW, because they are outputs of that mapping.
Interpret geometrically: The range containing these outputs is described as an n-dimensional linear subspace of Rᵈ.
Different inputs can produce different recovered vectors, but the recovered outputs are contained in the same n-dimensional linear subspace determined by UW.
Mistakes to Avoid
Treating W and U as independent choices
The solution requires W = Uᵀ, so W is not an unrelated second choice.
Fix:
State both the orthonormality condition UᵀU = I and the transpose relationship W = Uᵀ.Calling Wx the recovered vector
Wx is the compressed representation. The recovered vector is produced after U acts on that representation.
Fix:
Use UWx for the recovered vector and compare x with UWx.Describing the subspace as belonging to one input only
The subspace is the collection of all possible outputs of UW as x varies.
Fix:
Relate the subspace to the range of the fixed mapping UW.Forgetting the objective
PCA chooses the matrices to minimize total squared distance between original vectors and recovered vectors.
Fix:
Connect the compression and recovery stages to the total squared reconstruction-distance objective.
Check Your Understanding
Explain the full path of an input vector x through PCA using W and U. Then state which two expressions define the structure of the solution and describe what set contains all possible outputs UWx.
Hints
- Start with the compressed representation Wx.
- Identify the recovered vector after U is applied.
- Include both UᵀU = I and W = Uᵀ.
- Use the range of UW to describe the collection of recovered outputs.
What do you think happens?
If U and W remain fixed while x changes, do the recovered outputs remain in the same geometric collection?
Reveal answer
Answer: Yes, because every output is produced by the same mapping UW.
With U and W fixed, all outputs UWx belong to the range of UW, which is described as an n-dimensional linear subspace of Rᵈ.
Key Takeaways
- PCA chooses U and W to minimize total squared distance between original vectors and their recovered versions.
- W compresses x into Wx, and U maps that representation back to the recovered vector UWx.
- The solution has orthonormal columns in U, expressed as UᵀU = I.
- The compression matrix is the transpose of U, expressed as W = Uᵀ.
- As x varies, the outputs UWx form the range of UW, described as an n-dimensional linear subspace of Rᵈ.
Key Takeaways
- PCA is a reconstruction optimization: it minimizes total squared distance between original vectors and recovered vectors.
- The recovered vector is UWx because W compresses x and U uses the compressed representation to recover it.
- The solution satisfies UᵀU = I and W = Uᵀ.
- The collection of all outputs UWx is the range of UW, an n-dimensional linear subspace of Rᵈ.