Dataset Engineering: Curating, Generating, and Annotating Data

Imagine someone asks you to prepare training examples for a system. In one task, each example is an email and needs a single label. In another task, each example is a request to write a long essay. Both jobs involve putting labels on data, but the work they demand is very different. Dataset engineering is the discipline that covers how data is curated, generated, and annotated so that an AI system can learn from it. This article follows how the labeling problem changes as the task moves from a short menu of answers to an open-ended one.

The clearest example of a close-ended task is spam classification. The model has only two possible outputs, spam and not spam, so the full label set is known before any data is collected. Most traditional machine learning use cases fall on this close-ended side. Because the answer space is small and fixed, the question of what the correct label is has a straightforward answer for each example.

Foundation models change this picture. Their outputs are open-ended, so they are not limited to a short list of values. A request to write an essay can be answered in countless ways, and many of those answers may be acceptable. This means the question of what counts as a correct label becomes much less obvious. Before you can label data, you have to know what the set of possible correct outputs looks like, and for open-ended tasks that set is not a short menu.

Read the diagram as a question about the output space. On the close-ended side, the labeler is choosing one value from a known set of two. On the open-ended side, the labeler is facing a set of possible responses with no fixed boundary, and the diagram shows several of them as candidates. The difference is not about how many examples there are. It is about how many outputs count as correct.

Annotation means deciding what the correct label is for an example. The source points out that this decision is what makes annotation hard. For a close-ended task, the decision is a selection from a short menu. Two annotators who understand the task will usually land on the same choice, because the options are narrow and the criteria are clear.

For an open-ended task, the annotator must produce or judge something whose quality depends on many factors. There is no single menu entry to select. The annotator has to weigh several qualities of a response, and the decision is far less constrained. This is the reason the source concludes that data annotation is a much bigger challenge for AI engineering.

The source identifies a second difference between the two engineering settings: the kind of data each one works with. Traditional ML engineering works more with tabular data, meaning data organized into rows and columns with a defined structure. Foundation models work with unstructured data, such as free-form text and other content that does not arrive in a fixed grid of fields.

Because unstructured data has no built-in schema, the steps used to prepare it look different. Tabular data already tells you where each value lives. Free-form text does not, so extra preparation steps are needed before a model can use it well. This is the reason AI engineering adds data manipulation work on top of what traditional ML engineering already does.

In AI engineering, data manipulation is described as being about four activities: deduplication, tokenization, context retrieval, and quality control. These four activities form one umbrella category. The diagram below shows how they relate to that single idea.

Read the map as a list of responsibilities rather than a pipeline. Each of the four activities is a distinct kind of work, and the subsections below take them one at a time.