Data Quality and Data Diversity

Anyone preparing data for finetuning eventually asks the same question: how many examples do I need? A common instinct is that the answer is a large number, and that a handful of examples is wasted effort because a model that small cannot possibly learn anything useful from it. This article examines that instinct. It also looks at the other forces that decide how much data a project actually requires, beyond the quality and diversity of the examples themselves.

Keep your answer in mind. The reasons behind the answer depend on three things beyond the examples themselves: the finetuning technique you choose, the complexity of the task, and how well the base model already performs before any finetuning begins. Each of these shifts the number of examples you need up or down, and the three interact with each other.

The belief that a small dataset cannot help rests on treating data as the only lever. A failed finetuning run with few examples is easy to blame on the number of examples, but many other things can produce the same disappointing result. The source names three common culprits: a learning rate that is too high or too low, data that contains errors, and prompts that were poorly constructed.

The technique you choose is one of the biggest determinants of data requirements. Full finetuning updates the model as a whole and is expected to deliver the strongest performance, but it demands far more data than parameter-efficient finetuning, often called PEFT. LoRA is the example the source uses for a PEFT method. The gap is not small: full finetuning needs orders of magnitude more examples than PEFT methods like LoRA.

The practical reading is direct. If your collection is in the tens of thousands to millions of instruction-response pairs, full finetuning becomes worth attempting. If you only have a few hundred or a few thousand examples, PEFT is the more likely fit. The choice of technique therefore follows from the volume you can realistically assemble, not the other way around.

Technique is only one lever. Two more forces move the example count: how demanding the task is, and how close the base model already is to the performance you want. Both are easier to reason about once you see how they connect to the number of distinct patterns the model must learn.

Notice what the experiment does and does not claim. It shows a pattern across two scales in one setting. It does not establish that every base model converges at every data volume. Treat it as evidence for the direction of the effect, which is that a better starting point reduces the data you need and that abundant data can offset differences in the starting model.

The pilot step is the cheapest way to make the decision. A well-crafted set of about 50 examples costs little to assemble. If performance clearly improves, that is a signal that a larger dataset is likely to help more. If it does not, the sensible move is to examine hyperparameters, data quality, and prompts before committing to a large collection effort.

Read the flow as a sequence of checks. The pilot tells you whether more data is likely to pay off. The volume you can realistically assemble then decides between full finetuning and PEFT. A flat pilot sends you back to diagnosis rather than straight to collection, because a flat result has several possible causes.

The following mistakes come up when teams reason about small datasets. Each one treats the example count as the only variable, or ignores a factor the source identifies as shaping the decision.