Classical Machine Learning Algorithms
Feature engineering transforms raw data into more meaningful features before modeling.
From Raw Data to Useful Input
A machine learning model does not always receive data in the form that makes its task easiest. Feature engineering addresses this problem by transforming raw data into more meaningful features before the data enters the model. The transformation is designed before modeling rather than learned by the model itself.
Feature engineering is the use of non-learned transformations to turn raw data into more meaningful features before modeling.
Why Representation Changes Difficulty
Feature engineering can make a problem easier by expressing it in a simpler way. The model still has a learning task, but the input now highlights information that matters for that task. A good transformation can change a difficult machine learning problem into a much simpler one.
A Hypothetical Representation Change
Imagine a learning task whose raw inputs contain detailed observations, but the task depends mainly on whether each observation belongs to one of two broad conditions.
Inspect the raw data: The raw representation contains more detail than the model may need for this task.
Identify task-relevant information: Understanding the problem suggests that the distinction between the two broad conditions is the meaningful part.
Create a feature: A non-learned transformation represents each observation using a feature that captures the relevant condition.
Give the feature to the model: The model now works with an input representation that expresses the task more simply.
The example illustrates why feature engineering is more than rearranging data: it uses knowledge of the problem to produce a representation that can make the learning task simpler.
The Classical Workflow
Before deep learning, feature engineering was critical to the success of classical shallow algorithms. These algorithms did not have hypothesis spaces rich enough to learn useful features by themselves. As a result, the way data was presented to the algorithm was an essential part of building the solution.
In this historical workflow, the algorithm was responsible for learning from the supplied features, while feature engineering carried much of the responsibility for making the raw data suitable for learning. This is why feature engineering was not a minor preparation step for classical machine learning; it was central to the algorithm's success.
Classical and Deep Learning
Deep learning changes the balance of responsibilities. Deep learning models can automatically extract useful features from raw data, so modern deep learning reduces the need for most manual feature engineering. The features can emerge through the model rather than being designed entirely before the model receives the data.
| Aspect | Classical shallow algorithms | Deep learning |
|---|---|---|
| Feature creation | Useful features generally need to be designed before modeling | Useful features can be extracted automatically by the model |
| Role of representation | How data is presented is essential to success | The model can learn useful representations from raw data |
| Value of manual features | Historically critical | Reduced in many cases, but not eliminated |
Mistakes About Feature Engineering
Treating feature engineering as ordinary data rearrangement
Feature engineering is intended to create more meaningful features and make the problem easier, not merely to move values around.
Fix:
Start by understanding which parts of the raw data matter for the task and how they should be represented.Assuming that deep learning makes feature engineering completely irrelevant
Deep learning reduces the need for manual feature engineering but does not eliminate its value.
Fix:
Consider whether good manual features could produce a more elegant solution, use fewer resources, or work with less data.Assuming a classical algorithm can always discover useful features from raw data
Classical shallow algorithms historically did not have hypothesis spaces rich enough to learn useful features by themselves.
Fix:
Treat the input representation as a central part of the classical machine learning solution.
Check Your Reasoning
A classical shallow algorithm receives raw data that does not express the task clearly. Explain what feature engineering should do before training and why this preparation was historically important.
Hints
- Describe the change from raw data to meaningful features.
- Explain how a better representation can simplify the learning problem.
- Connect the answer to the limited ability of classical shallow algorithms to learn useful features by themselves.
Compare the following two situations: a classical shallow algorithm receives manually engineered features, while a deep learning model extracts useful features automatically. In your comparison, identify what remains valuable about manual feature design in the second situation.
Hints
- Focus on where feature extraction occurs.
- Mention the possible benefits of a more elegant solution, fewer resources, or less data.
Key Takeaways
- Feature engineering transforms raw data into more meaningful features before the data enters a model.
- A well-designed representation can turn a difficult machine learning problem into a simpler one.
- Designing useful features requires understanding which parts of the raw data matter for the task.
- Feature engineering was historically critical for classical shallow algorithms because they could not learn useful features by themselves.
- Deep learning can extract features automatically, but manual features can still provide elegant, resource-efficient, and data-efficient solutions.
Key Takeaways
- Feature engineering is a non-learned transformation performed before modeling.
- Its purpose is to express raw data in a form that makes the learning task more meaningful and often simpler.
- Classical shallow algorithms depended heavily on engineered features because they could not generally learn useful features on their own.
- Deep learning reduces the need for manual feature engineering by extracting features automatically.
- Manual feature design remains useful when it creates an elegant solution, reduces resource needs, or allows learning from less data.