Understanding Univariate Statistics for Feature Selection

When a dataset has many features, a natural first question is which of them deserve to be kept. One family of answers works by looking at each feature by itself. Instead of asking how a group of features behaves together, you ask a narrower question: for this one feature, is there a statistically significant relationship with the target? You repeat that question for every feature, then keep the ones whose relationship is strongest.

Methods built this way are called univariate statistics. The word univariate signals the key constraint of the method: each test considers only one feature at a time. Almost everything about the approach follows from that constraint, including its practical strengths and its blind spots. This article traces both, step by step.

Picture the procedure as a pipeline. Every feature enters the test separately. For each one, the test produces a single answer about that feature's link to the target. The answers are then sorted by how confident they are, and the top of the list is kept. No feature is compared with another feature during the test, and no model is trained along the way.

Notice what the test is and is not doing. It does not build a predictive model, and it does not try out different combinations of features. It produces, for each feature, an answer to one question about that feature's link to the target. The selection step then sorts the features by how confident each answer is and keeps the strongest ones.

The defining weakness of univariate tests is that a feature will be discarded if it is only informative when combined with another feature. A single feature can look like noise while a pair of features predicts the target perfectly. To see how this happens, consider a small invented dataset with two binary features, a and b, and a binary target y. Each row lists a, b, and then y. The target is 1 exactly when the two features differ. This pattern is known as exclusive or.

The left column of the comparison is what a univariate test sees. Each feature alone gives the same average, so both look uninformative. The right column is what the combination reveals. Once a and b are considered together, every pair of values determines y. The univariate procedure never looks at that right-hand column, which is why the feature pair can be discarded.

Univariate tests are often very fast to compute, and they do not require building a model. Because no model has to be trained, the cost of the selection step stays low even when there are many features to examine. Each feature is tested once against the target. This makes the approach a practical first pass over a large set of candidate inputs.

The same tests are also independent of whatever model you apply after feature selection. The scores they produce do not depend on which learning method you choose later. You can therefore run the selection once and then experiment with different downstream models without repeating the feature analysis for each one. The diagram below shows where this filter sits in the overall workflow.

What gets passed from the filter to the model is only the list of selected features. The filter does not hand over a trained model or a set of combined features. Because the scores were computed without reference to the model, the same selected set can feed several different models in turn.

The univariate approach has a specific name in classification problems, where the target is a category rather than a number. There, univariate statistics is also known as analysis of variance, usually abbreviated ANOVA. When you meet the term in a feature selection setting, read it as the same idea described above: an individual test of each feature's relationship with the target, followed by keeping the features with the strongest relationships.