Developing a Model that Overfits
Think of machine learning as a workflow of connected decisions, not only as model construction.
The Workflow Before the Model
Machine learning is often described as building a model, but model construction is only one part of the work. A stronger way to think about a machine learning project is as a workflow of connected decisions. The team first defines the real-world problem, its inputs, and its target output. It then prepares the data, decides how success will be measured, develops a model, and considers whether the model is fitting the training data too closely.
Each decision affects the next one. If the problem and target output are unclear, the data and evaluation plan are unclear as well. If the evaluation plan is missing, the team cannot determine whether the model is successful.
From Raw Data to Model Inputs
After the real-world problem has been defined, the available data must be made suitable for the machine learning task. Preparation is not one isolated action. The workflow can include cleaning the available data, transforming it, preparing features, and separating data into training and evaluation sets. Together, these stages create a usable path from raw information to a model.
Feature preparation belongs inside this workflow rather than being treated as an unrelated technical detail. The features are the prepared inputs that give the model information to work with. The purpose of the preparation stage is to make a usable path from the raw information to the model.
Defining Success Before Evaluation
A measure of success is the chosen basis for judging whether a machine learning result is successful during evaluation.
A model cannot be judged without a measure of success. The measure should be chosen as part of defining the machine learning problem, not added after the model already exists. It determines what the project treats as a successful result during evaluation.
A Team With Raw Data but No Evaluation Plan
A team wants to create a machine learning solution for a real-world task. It has collected raw data, but it has not stated the target output, selected a measure of success, prepared the data, or considered overfitting.
Define the task: State the real-world problem, identify the inputs, and specify the target output before judging any model.
Choose success: Select the measure that will determine whether the result is successful during evaluation.
Prepare the data: Move the raw data through cleaning, transformation, feature preparation, and separation into training and evaluation sets as appropriate for the task.
Develop and inspect the model: Develop the model using the prepared data, then check whether it has become closely fitted to the training data rather than solving the broader problem.
The team should treat the project as a connected sequence of decisions rather than beginning with model construction.
Tracing an Overfit Model
Overfitting occurs when a model becomes closely fitted to the data used for training. A strong fit to training data is not automatically the same as solving the broader machine learning problem. The danger becomes visible when the model's close fit to the training data is considered separately from how well it performs on data used for evaluation.
The important comparison is not simply whether the model can fit the training data. The workflow must also ask whether that fit represents success on the broader task. Separating training and evaluation data makes it possible to inspect this distinction during evaluation.
What do you think happens?
A model fits its training data very closely. Does that alone show that the machine learning problem has been solved?
Reveal answer
Answer: No, because close training fit can be overfitting.
The model can be closely fitted to the training data without solving the broader machine learning problem. Evaluation is needed to judge success using the chosen measure.
Applying the Sequence to a New Task
The same reasoning can be transferred to a different machine learning scenario. The details of the real-world task may change, but the connected decisions remain: define the problem, choose how success will be measured, prepare the data and features, separate training from evaluation, develop the model, and inspect the possibility of overfitting.
Planning a Machine Learning Solution for Delivery Delays
A team wants to create a machine learning solution related to delivery delays. The team has collected raw data but has not yet planned the workflow.
Define the problem: State the real-world task, identify which information will be used as inputs, and specify the target output the solution should produce.
Choose a measure of success: Decide how the team will judge whether the result is successful before evaluating models.
Prepare the information: Clean and transform the raw data, prepare useful features, and separate the available data into training and evaluation sets.
Develop the model: Use the prepared inputs to develop a model, while remembering that a close fit to training data can become a warning sign.
Evaluate the result: Use the selected measure of success to judge the result and consider whether the model's training fit is hiding poor performance on the broader task.
The team has converted an undefined modeling task into a connected workflow of problem definition, preparation, modeling, and evaluation.
A team has raw data and wants to build a model immediately. Write the order of decisions the team should make before treating the model as successful. Include the target output, data preparation, the measure of success, and the risk of overfitting.
Hints
- Begin with the real-world problem, inputs, and target output.
- Place the measure of success before judging model results.
- Include cleaning, transformation, feature preparation, and separation into training and evaluation sets.
- Explain why a close fit to training data is not enough.
Mistakes in the Workflow
Starting with model construction before defining the real-world problem
Without a defined task, the later preparation and evaluation decisions have no clear purpose.
Fix:
Define the problem, inputs, and target output before evaluating models.Treating data preparation as a single action
Preparation is a connected path from raw information to a model, not one isolated step.
Fix:
Consider the full preparation process required to make the data suitable for the task.Choosing a measure of success only after seeing model results
The measure of success helps define what the project treats as a successful result.
Fix:
Choose the measure as part of defining the machine learning problem.Assuming a close training fit proves the problem is solved
A model can fit training data closely without solving the broader machine learning problem.
Fix:
Use evaluation data and the chosen measure of success to examine the result and consider overfitting.
Keep asking what decision the current stage enables. Problem definition identifies the task, preparation creates usable inputs, the success measure defines the judgment, and evaluation reveals whether a close training fit may be overfitting.
Workflow Check
- Machine learning is a workflow of connected decisions, not only the act of building a model.
- The workflow begins by defining the real-world problem, its inputs, and its target output.
- Raw data may pass through cleaning, transformation, feature preparation, and separation into training and evaluation sets.
- A measure of success must be chosen before results can be judged meaningfully.
- Overfitting is the danger that a model becomes too closely fitted to training data without solving the broader machine learning problem.
Key Takeaways
- Treat machine learning as a connected workflow of problem definition, preparation, evaluation, feature engineering, and model development.
- Define the target output and inputs before judging a model.
- Prepare raw data through cleaning, transformation, feature preparation, and separation into training and evaluation sets.
- Choose a measure of success as part of defining the problem.
- Recognize that a close fit to training data can be overfitting rather than evidence that the broader task has been solved.