Instruction Finetuning Data

Dataset design rests on three core criteria that apply across training phases: quality, coverage, and quantity.

Programming

Two Candidate Training Sets

Building the data for a model can feel like a vast and messy task. The reasoning behind it, however, is simpler than it first appears. A useful way to begin is to ask what you want the model to be able to do, and then to build a collection of examples that demonstrates exactly that behavior. The dataset is designed backward from the behavior you want, not forward from whatever files happen to be available.

What do you think happens?

Imagine two candidate training sets for the same model. The first is very large but full of errors, inconsistent labels, and off-topic entries. The second is much smaller, but each example has been carefully checked. Which set would you expect to produce the better model, and why?

The very large set, because more examples always help
The smaller, carefully checked set, because each example is trustworthy
Neither, because the size and quality of data do not affect a model

Reveal answer

Answer:

The smaller, carefully checked set is the one expected to produce the better model.

A small amount of high-quality data can outperform a large amount of noisy data, so quantity alone is not enough. The rest of this article explains the criteria that make this true.

The rest of this article follows one question: how do you judge whether a dataset is good enough to train on? The answer begins with three criteria that stay the same no matter what kind of training you are doing.

The Three Core Criteria

Dataset design rests on three core criteria: quality, coverage, and quantity. Each one answers a different question about the same collection of examples. Quality asks whether each example is good enough to learn from. Coverage asks whether the examples span the range of situations the model will need to handle. Quantity asks how much material is available to learn from in total.

Criterion	Question it answers	What a failure looks like
Quality	Is each example good enough to learn from?	Errors, inconsistent labels, or off-topic entries
Coverage	Do the examples span the situations the model must handle?	Important situations are missing from the collection
Quantity	How much material is available in total?	Too little material to learn from

The three criteria and the question each one asks of a dataset

Why Less Data Can Win

A small amount of high-quality data can outperform a large amount of noisy data. The reason lies in what each example contributes. If an example contains an error or an inconsistent label, the model is being taught something wrong, and adding more such examples adds more wrong lessons. A smaller set in which each example has been carefully checked gives the model a cleaner signal to learn from.

How a small carefully checked dataset and a large noisy dataset differ on each criterion. Coverage must still be considered for both, because the criteria apply to every dataset.

Worked example

Comparing two candidate training sets

Two training sets are proposed for the same model. Set A has a very large number of examples, but many contain errors and inconsistent labels. Set B has far fewer examples, and each one has been carefully checked. Which set should you expect to perform better, and what criteria justify the answer?

1
Check quality
Set A contains errors and inconsistent labels, so many of its examples are not good enough to learn from. Set B's examples have each been checked, so they pass the quality question.
2
Check quantity
Set A wins on quantity by a wide margin. Quantity alone, however, does not settle the question.
3
Check coverage
Both sets must still span the situations the model needs to handle. If Set B lacks important situations, its small size becomes a weakness on the coverage criterion, and that must be weighed against its quality advantage.
4
Weigh the criteria together
Because a small amount of high-quality data can outperform a large amount of noisy data, Set B is the better bet provided its coverage is adequate. The decision depends on all three criteria, not on quantity alone.

Result

Set B, the smaller carefully checked set, is the one to prefer when its coverage is sufficient. The large set's size cannot compensate for its errors.

How the Training Phase Changes the Data

The data you need depends on more than your application. It also depends on the training phase. Three phases are named: pre-training, instruction finetuning, and preferred finetuning. Pre-training requires different data from instruction finetuning, and both differ from preferred finetuning. The contents change from phase to phase, but the lens you use to judge them does not. Each phase is still evaluated through quality, coverage, and quantity.

Moving from one training phase to another changes the content of the data, while the three criteria used to judge it remain the same.

In the real world

Suppose a team moves from preparing data for one phase to preparing data for another. The team does not need a new set of judging rules. It asks the same three questions again: are the examples good enough, do they span what the model must handle, and is there enough material. Only the examples themselves change.

Synthetic Data and the Evaluation Gate

Because high-quality data is hard to acquire, many teams turn to synthetic data, meaning data produced programmatically. Generating data this way has long been a goal, but it became a practical solution for many more use cases once AI could create realistic, complex data. Synthetic data is practical for many uses, yet it must be evaluated for quality before it is used to train a model.

The order required for synthetic data. Generation comes first, but generated output does not feed training directly; it must pass an evaluation step before training.

Evaluating AI-generated data is just as tricky as evaluating other AI outputs. People are more likely to use generated data when they can reliably evaluate it. The practical lesson is that a generation method is only as useful as your ability to judge its output. If you cannot check the data, you have little basis for trusting it.

The same asymmetry appears across dataset creation. Generating data can be automated to a large degree, but verifying that the generated data is correct is harder still. Labeling data is hard, and writing the guidelines that tell annotators how to label is harder still. Automation helps most with the production step and least with the checking step.

Mistakes Teams Make with Dataset Size and Synthetic Data

Common mistake
Treating quantity as the only measure of a dataset
What it looks like
A team celebrates collecting a very large set and begins training without checking its contents.
Why it fails
Quantity is one of three criteria. A large amount of noisy data can underperform a small amount of high-quality data.
Do this instead
Judge the dataset on quality and coverage as well as quantity before deciding it is ready.
Common mistake
Designing the dataset from the available files instead of the desired behavior
What it looks like
A team gathers whatever examples are on hand and hopes the model will learn something useful from them.
Why it fails
The dataset is meant to be designed backward from the behavior you want the model to show.
Do this instead
Start by asking what the model should be able to do, then build examples that demonstrate exactly that behavior.
Common mistake
Using synthetic data for training without evaluating it
Why it fails
Generated output does not feed training directly. Synthetic data must be evaluated for quality first, and a generation method is only as useful as your ability to judge its output.
Do this instead
Place an evaluation step between generation and training, and check the generated examples against the three criteria.
Common mistake
Assuming the criteria change when the training phase changes

Checking a Dataset Against the Criteria

MEDIUM

A team has prepared synthetic instruction examples for a model. They have generated 50,000 examples, but they have not yet checked any of them. Describe, in order, what the team should do before using these examples to train the model, and name which of the three criteria each step addresses.

Hints

Recall where the evaluation step sits in the order from generation to training.
Consider which criterion asks whether each example is good enough to learn from.
Think about whether a large count alone tells the team the examples are ready.

Your attempt (not saved)

EASY

Explain, in two or three sentences, why the team should not decide the dataset is ready simply because it contains 50,000 examples.