Tackling Machine Learning Problems Effectively
Running a favorite algorithm first is usually a poor way to begin a data analysis, because machine learning is typically only a small part of a larger analysis and decision-making process.
Why the Favorite Algorithm Comes Too Early
Once you have learned a set of machine learning methods, a familiar impulse appears. You are holding a data problem, you already know an algorithm well, and the quickest route seems to be connecting the two and seeing what comes out. Many learners adopt this habit without noticing it. It feels productive, because the code runs and numbers appear on the screen.
This habit is a misconception. Jumping straight into a data problem by running a favorite algorithm, without first considering the larger problem, is usually a poor way to begin an analysis. The rest of this article explains why, and what should happen first instead. Before reading the explanation, make a prediction about a realistic situation.
What do you think happens?
A team is handed a fraud problem and has one algorithm they know well. Which should they do first?
Reveal answer
Define the goal and decide how success will be measured
The team should first decide what they are trying to accomplish, such as detecting fraudulent user transactions, and how they will know whether a solution works. The modeling choice follows from that goal. The team does not begin by opening the algorithm library.
Machine Learning as One Step in a Chain
The central correction is one of scale. A machine learning algorithm is usually only a small part of a larger data analysis and decision-making process. Choosing and running a model is one action inside a chain that begins with a question and ends with a decision that matters to the people or organization involved. To use machine learning effectively, you need to step back and look at the whole problem before you look at the model.
The diagram below shows where the modeling step sits. Data is gathered and explored before any model is involved, and the model's output only becomes useful when it feeds a decision. The algorithm sits near the middle-to-late part of the chain, not at the start.
Starting From the Question
The sequence of thinking begins before any model is built. The first decision is what kind of question you want to answer. You might be doing exploratory analysis, looking to see whether anything interesting appears in the data. Or you might already have a particular goal in mind. Both are legitimate starting points, but they call for different framing.
A goal gives a project its direction. Often you will start with a specific goal, such as detecting fraudulent user transactions, making movie recommendations, or finding unknown planets. A goal names the thing you are trying to accomplish, and that in turn shapes which data matters and what kind of result would count as useful. A goal such as finding unknown planets points you toward different data than a goal such as recommending movies, even if both use the same underlying method.
Defining Success Before Building
A goal is not yet a definition of success. Once you have a goal, the next step is to think about how you will define and measure success before building anything. Success has to be stated in a way you can check, and it has to connect back to the reason the project exists. Before building a system, consider what a successful solution would mean for the overall business or research goals.
Turning a fraud goal into checkable success
A team has been asked to help with detecting fraudulent user transactions. They have one algorithm they know well. Before they touch it, what should their goal and success definition look like?
- 1State the goal
The goal is to detect fraudulent user transactions. This names the thing the project is trying to accomplish.
- 2Connect the goal to the wider purpose
Ask what a useful result would do for the organization. Catching fraud matters because it protects users and the business, so the success definition should reflect that purpose rather than only a score.
- 3Make success checkable
Write down a measure the team can actually check after the system exists. The measure should be specific enough that two people reading it would agree on whether a result meets it.
- 4Let the model follow
Only now choose the algorithm. The goal and the success measure determine which algorithm choices are worth testing, so the algorithm serves the project.
The team has a goal, a checkable success definition tied to the purpose of the project, and a clear reason to choose an algorithm. The algorithm they already know well becomes one option to evaluate, not the starting point.
A movie recommendation goal
A project aims to make movie recommendations. How would the same goal-first thinking apply?
- 1Name the goal
The goal is to recommend movies that a user is likely to want to watch.
- 2Ask what success looks like for the people involved
Success should describe a useful experience for the viewer, such as recommendations users actually engage with, rather than only a technical number.
- 3Choose data and method afterwards
Once the goal and success are clear, decide what user information matters and which method fits that information.
The modeling choice comes after the goal and the meaning of success, exactly as in the fraud example.
Algorithm-First Versus Problem-First
The two workflows differ mainly in where they begin. An algorithm-first workflow starts with a method and looks for a problem it can be applied to. A problem-first workflow starts with a question and a goal, defines success, and chooses a method afterwards. The first feels faster, but the second keeps the project anchored to what it is for.
| Question | Algorithm first | Problem first |
|---|---|---|
| Where does the project start? | With a favorite method | With a question and a goal |
| What defines success? | Often left unclear | Stated and measurable before building |
| What does the model serve? | Whatever the method happens to produce | The goal and the decision it informs |
| Where does the algorithm sit? | At the beginning | Near the middle, after the goal is clear |
Two ways to begin a machine learning project
Mistakes When Starting With an Algorithm
Common mistake Treating a running program as evidence that the approach is right
What it looks likeA learner sees numbers appear on the screen and assumes the analysis is progressing.
Why it failsOutput appearing shows that code ran, not that it answers the question that matters.
Do this insteadCheck whether the output is tied to a stated goal and a measurable definition of success.
Common mistake Letting the algorithm define the project
What it looks likeA team chooses a method first and then decides the project is about whatever that method can predict.
Why it failsThe algorithm should serve the project. When it defines the project, the goal may drift toward what is easy to compute.
Do this insteadWrite the goal and success criteria first, then choose the algorithm that fits them.
Common mistake Assuming the model is the whole analysis
Why it failsMachine learning is usually only a small part of a larger analysis and decision-making process, so attention to the model alone leaves the surrounding steps unexamined.
Do this insteadLook at the chain from question to decision and ask where the model's output will be used.
Common mistake Stating success vaguely, such as wanting the system to work well, without a measure anyone could check
Applying the Problem-First Order
A small online shop wants to reduce the number of customers who cancel their orders. Before choosing any algorithm, write down the goal, one measurable definition of success, and one sentence explaining how a successful result would help the business. Then name the one algorithm you would consider only after those are written.
Hints
- Start by naming the thing the shop is trying to accomplish.
- A good success definition is one you could check after the system exists.
- The algorithm should follow from the goal and success, not the other way round.
Key Takeaways
- Running a favorite algorithm first is usually a poor way to begin a data analysis, because it tends to let the method define the project.
- Machine learning is typically only one step inside a larger analysis and decision-making process that starts with a question and ends with a decision.
- Begin by deciding whether you are exploring or pursuing a particular goal, such as fraud detection, movie recommendations, or finding unknown planets.
- Before building a system, define and measure success and consider what a successful solution would mean for the overall goals.
- Choose the algorithm only after the goal, success criteria, and impact are clear, so that it serves the project.