Data Manipulation in AI Engineering

Imagine you are building a system and someone asks you to prepare training examples for it. For one task, each example is an email that needs a single label. For another task, each example is a request to write a long essay. Both jobs involve putting human judgment into data, but they do not feel equally hard. This article explains why, and it uses that difference to introduce the data work that AI engineering adds on top of traditional machine learning.

The clearest illustration is spam classification. The model has only two possible outputs, spam and not spam, so the label set is fully known in advance. Most traditional machine learning use cases fall on this close-ended side. Because the answer space is short and fixed, the labeling task reduces to choosing one item from a small menu.

Foundation models change this picture because they are open-ended. Their outputs are not limited to a short list of values. A request to write an essay can be answered in countless ways, and no fixed list of correct answers exists. This means the question of what counts as a correct label becomes much less obvious.

The comparison shows the structural difference. A close-ended task has a short, fixed set of acceptable labels, so the labeler's job is to pick one. An open-ended task has no such set, so the labeler's job becomes judging whether a response is good, which is a different kind of decision.

Annotation requires an annotator to decide what the correct label is. For a close-ended task, that decision is a selection from a short menu, and two annotators who understand the task will usually land on the same choice. For an open-ended task, the annotator must produce or judge something whose quality depends on many factors, so the decision is far less constrained. Two careful annotators can reasonably reach different judgments, which is why the source concludes that data annotation is a much bigger challenge for AI engineering.

The source identifies a second difference between the two engineering settings: the kind of data each one works with. Traditional ML engineering works more with tabular data, meaning data organized into rows and columns with a defined structure. Foundation models work with unstructured data, such as free-form text and other content that does not arrive in a fixed grid of fields.

Because unstructured data has no built-in schema, the steps used to prepare it look different. A table already tells you which column holds which field. Free-form text does not, so the data work has to be designed for content whose structure is not given in advance. This is the reason AI engineering adds its own set of data manipulation activities.

In AI engineering, data manipulation is described as four activities: deduplication, tokenization, context retrieval, and quality control. The map below shows how these pieces relate to the single umbrella idea. It shows the four activities as parts of one category and does not claim a fixed order among them.

To see what deduplication changes, consider a small illustrative dataset of support records. Two records contain almost the same text, differing only by punctuation. Removing the near-duplicate leaves a dataset where each distinct record appears once.

In the before state, the password reminder appears twice. In the after state, it appears once and the refund summary is untouched. The point is that deduplication changes the dataset's contents while leaving the distinct information in place.

Because quality control has these specific responsibilities, the quality question for an open-ended dataset is wider than whether an annotator picked the right option. It also asks whether the data should be in the dataset at all.