Problem Formulation in Machine Learning
Problem formulation asks you to slow down before either of those steps.
Why the Question Comes First
It is tempting to start a machine learning project by collecting data and choosing a model. Problem formulation asks you to slow down before either of those steps. Before you can phrase anything, you need to know what you are trying to answer, and then ask whether the data you collected can answer that question at all.
A question that the available data cannot address will not become a good machine learning problem, no matter how cleverly it is phrased. Formulation therefore depends on two things at once: a clear statement of the goal and an honest look at the data. Each decision made at this stage shapes everything that follows, including which model you can train, how you evaluate it, and whether anyone uses it.
What do you think happens?
A team has a dataset and a model ready. Should they train the model first and check whether the data fits the goal afterward?
Reveal answer
No, the goal and the data should be checked before training
The planning questions are treated as a checklist worked through in order. The first question asks what you are trying to answer and whether the collected data can answer it, and the early formulation decisions shape everything after them. A model can be technically sound and still answer the wrong question.
Tracing the Six Planning Questions
Before any model is trained, a project needs answers to a set of planning questions. Six key questions cover the question you are trying to answer, the data you have, how the problem is phrased, the features you extract, how success is measured, and how the solution fits into a larger product. Think of them as a checklist you work through in order, where each item can expose a different way the project can fail.
- Question 1: What am I trying to answer, and can the collected data answer it?
- Question 2: How should that question be phrased as a machine learning problem?
- Question 3: Have I collected enough data to represent the problem?
- Question 4: Which features did I extract, and will they enable the right predictions?
- Question 5: How will success be measured?
- Question 6: How will the machine learning solution interact with other parts of the product?
Notice the order. Questions one and two concern the goal and its phrasing, questions three and four concern the data and features, and questions five and six concern evaluation and integration. A failure in an early question can make later work wasted, which is why the checklist is read in sequence rather than as a loose collection of tips.
Checking Whether the Data Can Answer the Question
The feasibility check is the first gate in the process. You state the goal, then ask whether the data you have can answer that goal. If the answer is yes, you move on to phrasing the problem. If the answer is no, the goal or the data has to change before anything else happens. The process loops back rather than pushing forward with a mismatch.
Phrasing a Question as an ML Problem
Once the data can answer the goal, the practical question has to be restated in a form a model can learn from. This means deciding what the model receives as input, what it should output, and how known correct answers will be recorded in the collected examples. The phrasing step turns a loose business question into a concrete learning task.
Restating a Retention Question
A subscription business asks: Which customers are likely to leave, and can we do something about it? Phrase this as a machine learning problem.
- 1Name the practical goal
The business wants to identify customers who will cancel, so it can act before they leave.
- 2Choose the target output
The target is whether a given customer cancels within a chosen future window, which the model should predict.
- 3Choose the inputs
Features are measurable facts about each customer, such as how recently they last logged in or how many support tickets they opened.
- 4Identify the labels
Past customers whose outcomes are already known supply the labels, showing who cancelled and who stayed.
- 5Check the answer against the data
If no past records show who cancelled, the phrasing cannot be trained, and the question must be revisited.
The practical question becomes: given customer features at a point in time, predict whether that customer cancels within the chosen window, using past customers with known outcomes as labels.
Judging Whether the Data Represents the Problem
The third question asks whether you have collected enough data to represent the problem you want to solve. The key word is represent. The question is not whether you have a large amount of data in general. It asks whether the data you hold stands in for the real problem. A dataset can be big and still fail to represent the situation it is meant to describe.
Sufficiency is judged against the problem, so the same dataset can be enough for one question and not enough for another. When you answer this question, compare what the data covers with what the problem requires, and be honest about gaps before you commit to training. If the situations you care about are missing from the collected records, a model trained on them has nothing reliable to learn those situations from.
Judging the Extracted Features
The fourth question has two checks that belong together. First, you must be able to say what the features are. Second, you must judge whether those features actually help produce the predictions the problem needs. Features that are easy to obtain but unrelated to the goal are a common way for a project to look busy without being useful. The importance of extracting the right features is the central point of this question.
| Aspect | Well-chosen features | Poorly chosen features |
|---|---|---|
| Relationship to the goal | Connected to the outcome being predicted | Easy to collect but unrelated to the outcome |
| Mapping to model inputs | Describe the real situation the model must reason about | Describe something adjacent that the model cannot use |
| Effect on the project | Make the predictions the problem needs possible | Make the project look busy without making it useful |
Judging features against the goal, using a retention example
Defining Success and Product Fit
The fifth question asks how you will measure success in your application. This should be answered before the model is built, not after. If success is not defined in advance, it is easy to judge a result by whatever looks favorable. The measure has to fit the application, so the team must decide what counts as a good outcome for the task at hand.
The sixth question asks how the solution will interact with other parts of your research or business product. A model rarely works alone. It receives inputs from one part of a system, and its outputs are used by another part. Considering this interaction early matters, because the requirements of the surrounding product can shape the data, the features, and the way success is judged.
Mistakes Beginners Make in Formulation
Common mistake Training a model before checking whether the data can answer the goal
What it looks likeA team trains a classifier on whatever records they have, then discovers the records never capture the outcome they care about.
Why it failsA question the available data cannot address will not become a good machine learning problem, no matter how it is phrased.
Do this insteadRun the feasibility check first and loop back to revise the goal or collect different data when the two do not agree.
Common mistake Assuming a large dataset is a representative dataset
What it looks likeMillions of records come from one region, but the product serves customers everywhere.
Why it failsThe question is whether the data stands in for the real problem, not whether it is large.
Do this insteadCompare what the data covers against what the problem requires and list the gaps before committing to training.
Common mistake Choosing features because they are easy to obtain
What it looks likeAn office parking spot is recorded for every customer because it is in the database.
Why it failsFeatures unrelated to the goal make a project look busy without making it useful.
Do this insteadFor each feature, state what it is and judge whether it helps produce the predictions the problem needs.
Common mistake Deciding how to measure success after seeing the results
What it looks likeA team picks whichever metric makes the first model look good.
Why it failsIf success is not defined in advance, a result is easy to judge by whatever looks favorable.
Do this insteadDefine the success measure before building the model, chosen to fit the application.
Common mistake Ignoring how the model's output will be used by the rest of the product until deployment time
Walking a Project Through the Checklist
Now trace one project through all six questions in order. The aim is to see how each answer narrows or changes the next one, and where a project would stop or loop back.
Tracing a Churn Project Through the Six Questions
A subscription business wants to reduce cancellations. The team has login records and support tickets for past customers, but no record of why customers left.
- 1Question 1: goal and data
The goal is to predict cancellations. The data can answer a cancellation prediction only if past cancellations are recorded, and here they are, so the goal passes the feasibility check.
- 2Question 2: phrasing
Restate the goal as predicting whether a customer cancels within a chosen window, using past customers' outcomes as labels.
- 3Question 3: representation
Check whether the records cover all customer regions and plan types. If one plan type is missing, the dataset does not represent the problem and the gap must be recorded before training.
- 4Question 4: features
Login recency and support ticket counts relate directly to the outcome and are kept. Any feature with no connection to cancellation is dropped even if it is easy to collect.
- 5Question 5: success
Before training, the team agrees on how a good result will be judged, so they cannot pick a favorable measure afterward.
- 6Question 6: product fit
The predictions must reach the retention team in time to act. This requirement can change which features are usable and how quickly predictions are needed.
The project proceeds only after the representation gap is recorded and the success measure is fixed. Each answer shaped the next, which is the purpose of working through the checklist in order.
Checking Your Formulation
A team wants to predict which job applicants will accept an offer. Their dataset contains only applicants who received offers, with no record of who declined to apply in the first place. Work through the six questions and state which question exposes the most serious problem, and what you would do before training.
Hints
- Start with question three: does the dataset represent the problem, or only part of it?
- Think about which applicants the model would never see during training.
- Consider whether the gap should be recorded, the goal revised, or different data collected.
For a project of your own choosing, write one sentence for each of the six questions. If you cannot answer one honestly, note it as an open risk before any model is trained.
Hints
- Be specific about what each feature is.
- Define the success measure in one concrete sentence.
Key Takeaways
- A machine learning project should begin by stating the question it must answer and checking whether the available data can answer it. A question the data cannot address will not become a good problem, however it is phrased.
- Phrasing a practical question as a machine learning problem means identifying inputs, the target output, and the labels that known outcomes provide.
- Data must represent the problem, not merely be large, and features must be judged on whether they enable the predictions the problem needs.
- Success should be defined before the model is built, and the solution should be considered in terms of how it interacts with the rest of the product.