Automatic Feature Selection
Binning, polynomials, and interactions have a large influence on linear models and naive Bayes models, which depend heavily on how the features are represented.
Why Transformations Do Not Help Every Model
What do you think happens?
If you bin a feature or add interaction columns to a dataset, should you expect the same improvement from every model you train?
Reveal answer
No, the same transformed dataset can help one model a lot and another model very little.
Binning, polynomial terms, and interactions reshape the data, but how much that reshaping matters depends on the model family. Linear models and naive Bayes models depend heavily on how the features are represented, so they are usually strongly affected. Tree-based models can often discover important interactions on their own.
Once you know that binning, polynomial terms, and interactions can reshape a dataset in useful ways, the natural next question is whether to apply these transformations to every model you train. The answer is no, and the reason reveals something important about how models differ. A transformation is not a universal upgrade. Its value depends on what the model can already do with the raw columns it is given.
This chapter is about automatic feature selection, which is the other side of the same question. Transformations can create many new features, and some of them help while others add noise. Choosing which features to keep, and deciding when a transformation is worth the effort, both depend on understanding how different model families respond to changes in representation.
How a Linear Model Sees Binned Features
Binning, polynomials, and interactions can have a large effect on how well a model performs on a given dataset. The effect is especially strong for less complex models. The two examples the source gives are linear models and naive Bayes models. For these models, the form in which the information is presented matters a great deal.
A linear model can only combine features in a straight-line fashion. Suppose the relationship between a feature and the target is curved, or depends on how two features combine. The model cannot capture that structure unless someone builds it into the input. Transformations are one way of doing exactly that. A bin, a polynomial term, or an interaction column is a new input that gives the linear model a piece of the curved or combined relationship it could not otherwise express.
This is the central reason transformations matter so much for linear models. The model's own flexibility is limited, so the work of representing a richer relationship is handed to the features themselves. A transformation that creates a useful new column can change the model's results substantially. The same transformation applied to a more flexible model may change very little.
Read the diagram as a change of representation rather than a change of data. The left side is one column that a linear model can only weight with a single straight-line slope. The right side is three columns, each of which is either on or off for a given row. The linear model can now give each group its own contribution, which lets it approximate a step-shaped or curved pattern that a single slope could never follow.
Tree-Based Models and Other Model Families
Tree-based models stand in contrast to linear models. According to the source, they are often able to discover important interactions on their own, and most of the time they do not require the data to be transformed explicitly. A tree-based model splits the data through a sequence of decisions. Through those decisions it can pick up combinations of features without someone creating an explicit interaction column first.
For a learner choosing a model, this has a practical consequence. Spending effort on manual interactions may give little return when the chosen model is tree-based. Time spent engineering features is usually better directed at the model families that depend on representation.
Some model families sit between the two extremes. The source names SVMs, nearest neighbors, and neural networks. These models might sometimes benefit from binning, interactions, or polynomials, but the implications are usually much less clear than they are for linear models. You should not expect a predictable gain from transformations with these models. Instead, treat a transformation as an experiment whose result has to be measured on held-out data.
The Cost of Adding More Features
Once you know how many new features transformations can create, it is tempting to keep creating them until the dataset has far more columns than it started with. The source warns against this. Adding more features makes all models more complex, and that increases the chance of overfitting.
Overfitting means the model learns patterns that belong specifically to the training data, so its performance on new data suffers. A model with many features has more ways to fit noise. Each added column, whether it is a raw feature, a binned version of a feature, a polynomial term, or an interaction, gives the model another route to match accidental patterns in the training rows. This is why the number of features is a real design decision rather than something to maximize.
The source offers a practical remedy. When you add new features, or whenever you work with a high-dimensional dataset, reduce the number of features so that only the most useful ones remain. Discarding the rest tends to produce simpler models, and simpler models often generalize better, meaning they perform more reliably on data they were not trained on.
Three Strategies for Choosing Features
The obvious follow-up question is how to tell which features are good. The source names three basic strategies, and each answers that question differently. Before looking at them individually, notice that all three are supervised. They use the target values, the thing the model is trying to predict, to decide which features to keep.
| Strategy | How it judges features | Uses a fitted model | Refits as features are removed |
|---|---|---|---|
| Univariate statistics | Scores each feature on its own against the target | No | No |
| Model-based selection | Uses the importance a fitted model assigns to each feature | Yes | Not required by the strategy itself |
| Iterative selection | Repeatedly removes features and refits the model | Yes | Yes |
The three basic strategies differ in how much they depend on a fitted model and whether they refit repeatedly.
Worked Example: Binning One Curved Feature
Helping a linear model with a binned feature
A linear model predicts a target from one feature, age. The relationship is not a straight line: the target is roughly flat for younger rows, rises in the middle, and flattens again for older rows. Decide how binning changes what the linear model can express.
- 1Start with the raw column
The model receives age as a single numeric column. Its prediction can change only by a constant amount for each unit change in age, which is a straight line.
- 2Identify what the straight line misses
The curved pattern cannot be captured by one slope. A single weight on age forces the same rate of change everywhere along the age range.
- 3Replace the column with bins
Split age into groups such as under 30, 30 to 60, and over 60. Each row now has a set of indicator columns, one per bin, that are either on or off.
- 4Let the linear model weight each bin
The model can now assign a separate contribution to each group. The combined prediction can rise and flatten in steps, which approximates the curved pattern better than the single slope did.
- 5Check the result on held-out data
Whether the binned version actually performs better must be measured on test rows, since the gain is not guaranteed for every dataset or every model family.
Binning gave the linear model the structure it could not build for itself. The transformation changed the representation, and that change, not any change in the data, is what improved what the model could express.
Fitting Selection on the Training Data Only
All three strategies use the target, so there is a trap to avoid. If you select features using the whole dataset and then evaluate the model on a test set, the test set has already influenced which features were kept. The test rows helped decide the features, so the test score no longer measures how the model generalizes to truly new data.
The source is direct about the correct order. Split the data into training and test sets first, then fit the feature selection only on the training part. The selected features are then applied to the test part, which remains a fair check of how the selected features generalize. The diagram shows where each step happens and where leakage would enter if the order were reversed.
Mistakes to Avoid When Selecting Features
Common mistake Applying the same transformations to every model
What it looks likeAdding interaction columns to a tree-based model and expecting a large gain.
Why it failsTree-based models can often discover important interactions themselves, so explicit transformations are usually not required for them.
Do this insteadDecide on transformations with the model family in mind, and measure whether each one helps.
Common mistake Assuming more features always help
What it looks likeKeeping every binned, polynomial, and interaction column that can be created.
Why it failsAdding features makes models more complex and increases the chance of overfitting.
Do this insteadReduce the feature set so that only the most useful features remain, which often produces simpler models that generalize better.
Common mistake Selecting features on the full dataset before splitting
What it looks likeRunning univariate scoring on all rows, then splitting into train and test.
Why it failsThe test rows influence which features are kept, so the test score is no longer a fair check.
Do this insteadSplit first, then fit the feature selection only on the training part.
Common mistake Treating a transformation as a guaranteed gain for SVMs, nearest neighbors, and neural networks
Try It: Choosing a Strategy and a Model
You are preparing a dataset with a continuous feature whose relationship to the target is curved. You plan to try a linear model and a tree-based model. Which model should you expect to benefit more from binning that feature, and why? Also explain where in the workflow the binning decisions and any feature selection should be fit.
Hints
- Think about how much flexibility each model has on its own, and which one the source says can discover interactions itself.
- Consider which steps use the target, and what must stay out of the test part.
Key Takeaways
- Binning, polynomial terms, and interactions have a large influence on linear models and naive Bayes models, which depend heavily on how features are represented. Tree-based models can often discover important interactions themselves.
- Adding features increases model complexity and the chance of overfitting, so reducing the feature set to the most useful ones can produce simpler models that generalize better.
- The three basic strategies are univariate statistics, which scores each feature on its own; model-based selection, which uses a fitted model's importance; and iterative selection, which repeatedly refits as features are removed.
- All three strategies are supervised, so feature selection must be fit only on the training part of the data, with the test part kept aside for a fair check.