Dataset Size and Model Reliability
A common first instinct when a model falls short is that the answer lies in the row count.
A Rule Built on Twelve Rows
Imagine you have studied a small table of customers and noticed a pattern that looks like a rule. The source material uses a boat-buying example. A table of 12 customer rows was used to develop this rule: if a customer is older than 45 and has fewer than 3 children, or is not divorced, then that customer wants to buy a boat. Before you rely on a rule like this, you need to ask how much evidence you have that the pattern is real. The answer depends heavily on how many rows you examined.
A common first instinct when a model falls short is that the answer lies in the row count. If the dataset gets bigger, the reasoning goes, then the model is allowed to get bigger too. This article tests that belief and then corrects it. The short version is that row count matters, but what those rows contain matters more.
What do you think happens?
You have a dataset of 12 rows. You copy the same 12 rows 1,000 times, so the file now has 12,000 rows. Does this justify using a more complex model?
Reveal answer
No. Copying the same rows makes the dataset longer but does not add new situations for the model to learn from.
The source states that duplicating data points, or collecting very similar data, adds little or nothing toward a more complex model. The question to ask is what the new rows show that the existing rows do not.
How Confidence in a Rule Grows
Think of a rule as a claim about the data. Each new row that matches the claim is another piece of evidence in its favor. When the rows are numerous and consistent with the rule, the claim becomes more believable. The reason is that the chance the same agreement appeared by accident shrinks as the number of supporting rows grows.
Return to the boat example. Suppose that 10,000 additional rows of customer data became available and every one of them agreed with the rule. The source states that you would be much more likely to believe the rule is a good one than you would have been based only on the original 12 rows. The diagram below traces how the state of your belief changes as the evidence accumulates.
Two Levers: More Data or More Tuning
When a model does not perform as well as you hoped, there are two broad levers to pull. One is to adjust the model by changing its settings, tweaking its parameters, and tuning it more carefully. The other is to gather more data. The source makes a practical point here: in many real situations you have some control over how much data you collect, and collecting more can be more beneficial than continued tweaking and tuning of the model.
| Question | Tuning the model | Collecting more varied data |
|---|---|---|
| What changes? | Settings and parameters of the existing model | The contents of the training dataset |
| Works on the same narrow dataset? | Yes, and it stays limited by that dataset | No, it widens the range of input cases the model sees |
| When is it the better investment? | When the model settings are the clear bottleneck | When the data covers too few situations for a more flexible model to be supported |
Two levers for improving a model that is not performing as hoped
Put these ideas together and the planning question changes. Rather than asking only whether to tune the model further, ask whether you can gather data that adds new variety. If the answer is yes, that data may do more for model quality than another round of tuning on the same narrow dataset.
Why Row Count Is Not the Real Limit
The source ties model complexity to the variation of inputs in the training dataset. A dataset is only as useful for supporting a complex model as the range of situations it covers. Model complexity is also not something to maximize. A model's complexity should be chosen in light of the amount of data available, so a larger dataset can support a richer model while a smaller one may not.
Duplicate Rows Versus Varied Rows
Consider two datasets with the same number of rows. The first contains 500 rows that are copies or near-copies of one kind of customer. The second contains 500 rows that cover different input cases, such as different ages, family sizes, and marital statuses. Both have 500 rows, yet only the second gives the model a wider range of situations to learn from. The comparison below shows why only the second supports a more complex model.
Deciding between two candidate datasets
Your team has a model trained on a modest dataset and wants a more flexible model. Two candidate datasets are available, each adding 500 rows. Dataset A is 500 near-copies of existing customers. Dataset B is 500 customers from age groups and household sizes the training data barely covers. Which one supports the more flexible model?
- 1Compare row counts
Both candidates add 500 rows, so row count alone cannot separate them. The source warns against stopping at this point.
- 2Ask what each set shows
Dataset A repeats patterns the model has already seen. Dataset B describes situations that are missing from the existing data.
- 3Connect variety to complexity
Because variety sets the complexity a model can safely use without overfitting, Dataset B is the one that supports a more flexible model.
Dataset B supports the more flexible model. Dataset A adds rows without adding variety, so it gives little or nothing toward added complexity.
Checking a Proposed Data Change
When someone proposes adding data, you can run a short sequence of checks before deciding whether it is likely to help. The first check is whether the proposed rows are new or copies. The second is whether they cover input cases the existing data lacks. The flowchart below sets out this sequence.
A colleague proposes adding 2,000 new rows that were collected by re-running the same survey question on the same group of customers a second time. Using the variety principle, would you expect this change to justify a more complex model? Explain your reasoning in terms of what the new rows show that the existing rows do not.
Hints
- Ask whether the new rows describe any input cases that the existing rows do not.
- Think about the difference between row count and variety of inputs.
Mistakes When Judging a Pattern
The reliability of a pattern depends on evidence, and several common errors come from judging that evidence badly.
Common mistake Treating a matching percentage as enough evidence
What it looks likeA rule matches 90 percent of rows in a 10-row table, and you conclude it is reliable.
Why it failsThe source states that matching percentages are not enough to judge a pattern. The number of supporting rows matters just as much.
Do this insteadReport both the percentage and the number of rows behind it before trusting the pattern.
Common mistake Believing that enough agreeing rows prove the rule
What it looks likeAfter 10,000 rows agree with the boat rule, you decide it holds for every customer.
Why it failsMore agreeing rows raise confidence in a rule but do not prove it is true in every case.
Do this insteadDescribe the rule as well supported by the evidence, and stay open to rows that break it.
Common mistake Assuming any extra data unlocks extra model complexity
What it looks likeThe dataset doubled, so the team switches to a much larger model.
Why it failsA larger dataset is not automatically a reason to use a more complex model. What matters is the variety of inputs it contains.
Do this insteadCheck what the new rows add in terms of input variety before increasing complexity.
Common mistake Counting duplicated rows as new evidence
What it looks likeA dataset is padded with copies of existing rows to reach a larger row count.
Why it failsDuplicates make the dataset longer but describe the same range of situations, and the source says they add little or nothing toward a more complex model.
Do this insteadRemove duplicates from the count and judge the dataset by the distinct input cases it covers.
Key Takeaways
- A rule becomes more believable as more data that complies with it accumulates, but this raises confidence rather than proving the rule.
- The variety of inputs in a training dataset sets how complex a model can safely be; row count matters mainly because it usually brings more variety.
- Duplicated or nearly identical rows add little or nothing toward a more complex model, because they do not widen the range of situations covered.
- In many real situations, collecting more varied data can be more beneficial than repeatedly tuning an existing model, so data collection is a real planning decision.
- Before acting on a proposed data change, check whether its rows are copies and whether they cover input cases the existing data lacks.