Data Sufficiency for Machine Learning

Imagine a team that has a large store of customer records and a strong wish to use machine learning. They gather the data, choose a model, and start training. Weeks later they have a model with impressive numbers, yet nobody can say whether it helps the business. The model may be technically sound and still answer a question nobody asked, or answer the right question with data that does not match the real situation. This article traces the early decisions that prevent that outcome.

Before a model is trained, a machine learning project needs answers to a set of planning questions. The source gathers these into six key questions. Together they cover the question you are trying to answer, the data you have, how the problem is phrased, the features you extract, how success is measured, and how the solution fits into a larger product.

The source presents these as a checklist that you work through in order. Each question exposes a different way a project can fail. A model can be technically sound and still answer the wrong question, or answer the right question using data that does not represent the situation. The diagram below shows the go or no-go logic that the first questions impose: a project only moves forward when the question, the data, and the phrasing hold up, and the later checks are passed before training begins.

The first question sets up the second. Before you can phrase anything, you need to know what you are trying to answer, and then ask whether the data you collected can answer that question at all. Formulation therefore depends on two things at once: a clear statement of the goal and an honest look at the data. A question the available data cannot address will not become a good machine learning problem, however neatly it is worded.

Phrasing is the act of restating a practical goal so that it names what the model receives and what it must produce. Once the goal is clear and the data has been checked against it, the practical question can be rewritten in terms of the inputs available, the output the business wants, and the prediction that connects them. The source does not give a fixed recipe for this rewriting, so the important habit is to make each part explicit before any training starts.

The third question asks whether you have collected enough data to represent the problem you want to solve. The key word is represent. The source does not ask whether you have a large amount of data in general. It asks whether the data you hold stands in for the real problem. A dataset can be big and still fail to represent the situation it is meant to describe.

Sufficiency is judged against the problem, so the same dataset can be enough for one question and not enough for another. When you answer the sufficiency question, compare what the data covers with what the problem requires, and be honest about gaps before you commit to training. The table below contrasts the two ways a team might judge its data.

The fourth question concerns features. Features are the values extracted from raw data that a model actually uses. The diagram below shows how raw data moves through extraction steps to become those features, and where a poor choice can enter the flow.

Two checks belong together here. First, you must be able to say what the features are. Second, you must judge whether those features actually help produce the predictions the problem needs. Extracting features that are easy to obtain but unrelated to the goal is a common way for a project to look busy without being useful. The source treats extracting the right features as the central point of this question.

The fifth question asks how you will measure success in your application. This is a question to answer before the model is built, not after. If success is not defined in advance, it is easy to judge a result by whatever looks favorable. The source identifies the need to measure success as an important fact, and asks you to determine the measure that fits your application.

The sixth question asks how the machine learning solution will interact with other parts of your research or business product. A model rarely works alone. It receives inputs from some part of a system, and its outputs are used by another part. The source asks you to consider this interaction early, because the requirements of the surrounding product can shape the data, the features, and the way success is judged.