Data Quality Checks Before Training

Imagine you are preparing data to train a model. A natural instinct is to collect as much of it as you can, on the reasoning that more examples means more learning. Before you commit to that plan, consider a question the source material raises directly: which of the following would you expect to train a better model?

This lesson is about how to judge data before training begins. The short answer from the source is that data is high quality when it helps you do your job efficiently and reliably. The rest of the lesson turns that idea into a concrete checklist and then shows how to apply it to individual records.

The first idea to hold onto is that quality is not a fixed property of a dataset. A dataset is good or bad only relative to what the model is supposed to do. The same collection of documents can be the right material for one project and useless for another.

Read the diagram as a dependency. The task comes first, and the judgment of quality is derived from it. The dataset does not carry a quality score around with it. Instead, the question 'is this good?' always has to be followed by 'good for what?'. The source's legal example makes this concrete. If the model's task is to answer legal questions as the law stands today, a legal dataset from the 19th century may not be relevant. If the model's task is instead to describe the legal system of the 19th century, that same dataset becomes highly relevant.

Quality checks are easiest to understand as a gate that a dataset passes through before it reaches training. The source lists six characteristics but does not prescribe an order for checking them. The sequence in the diagram below is one reasonable way to organize the work, and each stage asks the same kind of question: does this dataset meet this requirement, or does it need to be repaired or rejected?

Notice that the gate is not only about throwing data away. Some failures, such as formatting noise, can be fixed by cleaning the text, while others, such as material that is simply off-task, usually mean the record should be excluded. The source also reminds us that the checks are not an afterthought. Before creating data, you should think about what each characteristic means for your project, because the techniques used to produce data are aimed at producing exactly these qualities.

The source gives a common checklist. Data can be considered high quality if it has six characteristics: it should be relevant, aligned with task requirements, consistent, correctly formatted, unique, and compliant. The source also notes that some use cases carry extra requirements beyond these six. Treat the six as a default checklist to start from, then add task-specific checks when the project calls for them.

Now trace a single record through the checks. The example below is generated for illustration, and it follows the kind of scraped product review the source describes.

Notice what this example did not require. The review's meaning did not change, and no record had to be deleted. Some quality problems are repaired, not discarded. Other problems, such as an off-task document or personally identifiable information in a dataset that must not contain it, may require removing the record entirely.

Several readings of data quality sound reasonable but lead to poor decisions. The most important is the instinct the source warns about first: collecting as much data as possible.

The source also reports a tradeoff in the LIMA project. Its team fine-tuned a 65-billion-parameter Llama model on 1,000 carefully curated prompts and responses. Human annotators judged its answers equivalent to or strictly better than GPT-4's answers in 43 percent of cases. The same source notes that a model trained on that few examples was not as robust as product-grade models. A small, curated dataset can therefore be a strong choice for some goals while still falling short on others, and the checklist does not remove that tradeoff.

Work through this scenario before reading the summary. A team is building a model to answer short customer questions about a software product. Their training file contains the following three records, shown as they were stored.

The checks above can be used together as a sequence of questions. Each one returns to the task that the model must perform, because quality is judged against that task.