Preparing Unstructured Data for Models

A model can only learn from the examples it is shown. The quality and shape of those examples set a ceiling on what the model can do. Dataset engineering is the discipline that takes care of this material. It asks where training and adaptation data will come from, how that data should be prepared, and how each example gets its label or target. Before you can reason about a model's behavior, you need to know what went into it.

Before looking at the definitions, follow a single project through the work. The scenario below is an illustrative example written for this article to show how the three core activities often appear together in one project.

Together these three activities are the core of dataset engineering. They are not separate steps that never meet. A project might curate a collection of raw material, generate additional variants of it, and then annotate the full set, as the support-reply example above did. The diagram below shows how the three activities sit under the one term.

The difference that drives most of the extra work in AI engineering is the shape of the answer a task allows. In traditional machine learning, most use cases are close-ended: the model must choose among a fixed set of predefined values. Spam classification is the standard illustration, because the output is either spam or not spam and nothing else is possible.

Foundation models are open-ended. Their outputs are not restricted to a list fixed in advance, so a response might be a paragraph, a summary, or a piece of code. The diagram below sets the two kinds of task side by side.

Data manipulation covers the processes that turn collected data into something a model can use well. The source names four central processes: deduplication, tokenization, context retrieval, and quality control. Quality control in turn includes removing sensitive information and removing toxic data. The diagram below groups them around their shared purpose, which is to make collected data usable and trustworthy for a model.

Quality control is broader than the other three because it explicitly includes removing sensitive information and removing toxic data. The source places these removals in the same family of work because they protect users and shape the model's later behavior.

Tokenization is one of the four processes, and it is the one most tied to text. The source names it but does not describe the splitting rules, so this section offers only a general picture. Tokenization splits a piece of text into tokens, and each token is represented by a number the model can read. The example below is illustrative and was written for this article. The IDs are invented for teaching; real IDs depend on the tokenizer in use.

Read the map from left to right. The sentence the cat sat down becomes four tokens in order, and each token points to one numeric index. The model never receives the words themselves in this scheme; it receives the sequence of indices. Because tokenization is applied to unstructured text, it is one of the checks that has no equivalent in a plain table of fields.

Two differences separate AI engineering data work from traditional machine learning data work. The first is the output the task allows, which you saw with close-ended and open-ended tasks. The second is the form the data takes. Traditional machine learning engineering works more often with tabular data, organized into rows and columns with a defined set of fields. Foundation models work with unstructured data such as raw text, images, or audio, where there is no fixed grid of fields to rely on.

This is the reason data work becomes a much larger job in AI engineering. The same comparison also explains why the rest of the article's ideas, from annotation to manipulation, look different from the spam-classification examples that many introductions use.

The ideas in this article are easy to state and easy to misapply. The four mistakes below come up often when people first think about preparing data.