Quality Control for Training Data

A model can only learn from the examples it is shown. The quality and shape of those examples set a ceiling on what the model can do. Dataset engineering is the discipline that handles this material. It asks where training and adaptation data will come from, how that data should be prepared, and how each example gets its label or target. Before you can reason about a model's behavior, you need to know what went into it.

Dataset engineering covers three core activities. They often appear together in one project, so it helps to see how they connect before looking at each one separately.

The diagram shows that all three activities lead toward the same destination: the data a model uses to learn or adapt. Curating decides which existing material is worth keeping, generating fills gaps where examples are missing, and annotating turns raw examples into examples with a target the model can learn from.

The difference that drives most of the extra work in AI engineering is the shape of the answer a task allows. A close-ended task chooses from predefined values. In traditional machine learning, most use cases are close-ended. Spam classification is the standard illustration: the output is either spam or not spam, and nothing else is possible.

Foundation models are open-ended. Their outputs are not restricted to a list fixed in advance, so a response might be a paragraph, a summary, or a piece of code. The number of acceptable answers is no longer small and fixed, which changes what the data work must handle.

The source makes a direct comparison: judging whether an email is spam is much easier than writing an essay. The same asymmetry appears in annotation. Labeling a close-ended example means choosing from a short list, and two careful annotators will usually agree.

Annotating an open-ended example means deciding whether a free-form answer is good. That judgment involves many dimensions at once, such as accuracy, completeness, tone, and appropriateness. An annotator must weigh several qualities together rather than pick one value from a list, which is why annotating open-ended queries is described as a much bigger challenge for AI engineering.

A second difference concerns the form the data takes. Traditional machine learning engineering works more often with tabular data, organized into rows and columns with a defined set of fields. Foundation models work with unstructured data such as raw text, images, or audio, where there is no fixed grid of fields to rely on. AI engineering works more with unstructured data than the tabular data typical of traditional machine learning.

Data manipulation covers the processes that turn collected data into something a model can use well. The central work in AI engineering is deduplication, tokenization, context retrieval, and quality control. Each addresses a different kind of problem, so it helps to hold them as separate concerns even though they all serve the same purpose: making collected data usable and trustworthy for a model.

Follow one scraped customer chat record through the processes. The record starts as raw text, and each step changes what the collection holds. The walkthrough uses the illustrative order from the diagram, since the source does not require one fixed sequence.

The table below shows the effect of deduplication alone on a small collection. Only the exact repeats are removed, and each distinct record stays.

Quality control is broader than the other three processes, because it explicitly includes removing sensitive information and removing toxic data. Those removals protect users and the model's later behavior, which is why they belong in the same family of work as the other steps.