Principal Component Analysis (PCA)
The components are the shared building blocks.
Predicting the Required Property
Imagine you have a collection of data points, and you want to describe each one as a combination of a few shared building blocks. Each point becomes a recipe: how much of each building block it contains. This article is about one specific version of that idea, called Non-Negative Matrix Factorization, or NMF. It is also closely related to Principal Component Analysis, or PCA, which shares the same basic structure. NMF is an unsupervised learning algorithm that extracts useful features and can be used for dimensionality reduction.
What do you think happens?
Before reading on, predict the one property that the data must have for NMF to make sense as a recipe of building blocks.
Reveal answer
The data must be non-negative: every feature value must be greater than or equal to zero.
NMF requires both the components and the coefficients to be greater than or equal to zero, so it can only be applied to data where each feature is non-negative. A negative entry cannot be built from a sum of non-negative pieces.
Tracing a Point Back to Its Parts
NMF starts from an idea it shares with PCA. Each data point is written as a weighted sum of some components. The components are the shared building blocks. The weights, which the source calls coefficients, say how strongly each component contributes to a particular data point. Once the components and coefficients are known, the data point can be rebuilt by adding up the weighted components. In matrix shorthand, the data matrix $V$ is approximately the product of a component matrix $W$ and a coefficient matrix $H$, written $V \approx WH$.
Rebuilding a Message from Its Parts
Features are the counts of three words: hello, order, refund. Two components are given, and one message has coefficients 2 for component 1 and 3 for component 2. Rebuild the message as a weighted sum.
- 1Write the components
Component 1 is [1, 0, 0] (greeting words) and component 2 is [0, 1, 1] (order and refund words).
- 2Scale each component
Multiply component 1 by its coefficient 2, giving [2, 0, 0]. Multiply component 2 by its coefficient 3, giving [0, 3, 3].
- 3Add the scaled components
Add the two results position by position: [2 + 0, 0 + 3, 0 + 3] = [2, 3, 3].
- 4Check the sign
Every component value and every coefficient is zero or positive, so every contribution pushes the total upward. No part subtracts from another.
The message is rebuilt as [2, 3, 3], which matches the observed word counts.
The Non-Negativity Rule
The difference from PCA lies in the constraints. NMF requires both the components and the coefficients to be greater than or equal to zero. Nothing in the rebuilt data point can subtract from another part of it, so every contribution pushes the total in the same direction. The same rule applies to the input: NMF can only be applied to data where each feature is non-negative.
| Quantity | Required sign for NMF | Example in this article |
|---|---|---|
| Input features | Greater than or equal to zero | Word counts such as 0, 2, or 3 |
| Components | Greater than or equal to zero | [1, 0, 0] and [0, 1, 1] |
| Coefficients | Greater than or equal to zero | 2 and 3 |
Every quantity in the NMF model must be zero or positive.
Suppose a dataset records how often a set of words appears in each of several short messages. Every entry is a count, so none of them can be negative. This is the kind of data NMF can accept. Now suppose a second dataset records changes in temperature, where some readings are below zero. That dataset does not satisfy the requirement, because a negative entry cannot be built from a sum of non-negative pieces.
NMF Compared with PCA
NMF and PCA are closely related. Both write each data point as a weighted sum of components, and both can be used for dimensionality reduction. Where they differ is in what they optimize and which constraints they place on the parts. PCA seeks orthogonal components that explain maximum variance. NMF seeks non-negative components and coefficients instead.
| Feature | NMF | PCA |
|---|---|---|
| Shared structure | Each data point is a weighted sum of components | Each data point is a weighted sum of components |
| Use for dimensionality reduction | Yes | Yes |
| Constraint on components | Greater than or equal to zero | Orthogonal |
| Constraint on coefficients | Greater than or equal to zero | Not stated as a non-negativity rule in the source |
| Optimization goal | Non-negative components and coefficients | Components that explain maximum variance |
Both methods share a structure; the constraints and goals are what separate them.
Is Your Dataset a Fit?
Before choosing NMF, check the data in a fixed order. The first question concerns the sign of every input feature. Only if all features pass that check does the question of whether the parts can plausibly add up to each sample become relevant.
A dataset records the ratings of several movies, where each rating is between 1 and 5, and a second dataset records daily changes in a stock price, where some changes are negative. Which dataset can NMF accept, and why?
Hints
- Check whether every input value is zero or positive.
- A negative entry cannot be built from a sum of non-negative pieces.
- Ratings in this case never fall below 1.
Mistakes Learners Make with NMF
Common mistake Assuming NMF works on any numeric dataset
What it looks likeApplying NMF directly to temperature readings that include values below zero.
Why it failsNMF requires components and coefficients to be greater than or equal to zero, so it can only be applied to data where each feature is non-negative.
Do this insteadCheck every input feature for negative values first, and only proceed if all are zero or positive.
Common mistake Treating NMF and PCA as having the same constraints
What it looks likeBelieving that because both write data as a weighted sum, they optimize the same goal.
Why it failsPCA seeks orthogonal components that explain maximum variance, while NMF seeks non-negative components and coefficients.
Do this insteadCompare the constraints and objectives separately, as shown in the comparison table.
Common mistake Thinking a negative coefficient can make the rebuild work
What it looks likeUsing a coefficient of -1 to subtract a component from the total.
Why it failsNMF requires coefficients to be zero or positive, so nothing in the rebuilt point can subtract from another part of it.
Do this insteadUse only zero or positive coefficients, so every contribution pushes the total in the same direction.
Key Takeaways
- NMF writes each data point as a weighted sum of components, with coefficients stating how strongly each component contributes, similar to the structure used in PCA.
- NMF requires components and coefficients to be greater than or equal to zero, so it can only be applied to data where each feature is non-negative.
- PCA seeks orthogonal components that explain maximum variance, whereas NMF seeks non-negative components and coefficients.
- Before using NMF, check that every input feature is zero or positive; a dataset with negative entries, such as temperature changes below zero, is not a fit.