Synthetic Data Generation with Templates

Many people who start building a model assume the difficult part is the model itself. In practice, a team often spends more effort deciding what data the model should learn from. A dataset is rarely something you find ready-made. It usually has to be assembled from several places, cleaned, extended, and labeled before a model can use it.

The name for this assembly work is data acquisition. Like a supply chain, it has to be planned: you need to know which inputs you need, where they can come from, what they cost, and what state they will be in when they arrive. The goal is a dataset that is large enough, varied enough, and accurate enough for the task, collected in a way that respects user privacy and the rules that apply to the data.

Acquisition draws on four broad routes. Sourcing public data means finding datasets that other people have released. Purchasing proprietary data means paying a provider for data it holds and you do not. Annotating data means people add labels or other information to examples that already exist. Synthesizing data means new examples are generated from existing data or from templates.

These routes are not alternatives you choose between once. They are combined. A single dataset can contain records from several sources, and the same dataset can pass through annotation and synthesis more than once. The sourcing routes supply raw records, while annotation and synthesis change the records or add to them.

Read the diagram as a set of inputs and transformations. The two sourcing routes bring in records that already exist. Annotation and synthesis then either attach information to records or create new ones. A finished dataset is usually the result of several of these arrows, applied in some order, rather than a single one.

Of the four routes, the one that usually matters most is the data your own application produces. The reason is distribution. Your application's records reflect the kinds of inputs your users actually send, in the proportions they actually send them. Public and purchased datasets were gathered for someone else's purpose, so their match to your task is often only partial. Matching the distribution of the data you care about is hard to achieve with other sources, which is why application data is described as ideal for the task.

Application data can also feed back into the product. A useful pattern is a loop: users generate data, that data is used to improve the product, and the improved product generates more useful data. Teams often call this a data flywheel.

The design of a feedback system is a topic of its own, and it is covered later in the course.

Before you invest in creating data, check whether a suitable dataset already exists. Data marketplaces hold both open and proprietary datasets, and some may match your requirements closely. In many projects, though, the result is a combination of several partial matches rather than one perfect dataset. The useful habit is to search first, then decide which parts still have to be built.

Public data has a second requirement that is easy to forget. A public dataset is not automatically safe to use. Before using one, check its license, and try to understand where its contents came from. A license that permits commercial use can still cover a dataset that includes material collected under different terms. Public data is worth using, but it has to be inspected and validated like any other input.

Evaluation harness tools are another source of ready-made data. Some host benchmark datasets large enough for parameter-efficient fine-tuning. One example is the lm-evaluation-harness from EleutherAI, which hosts more than 400 benchmark datasets, averaging over 2,000 examples each. Such sources are still public data and follow the same check of license and provenance.

A dataset changes at every step of acquisition, and the easiest way to see how is to follow one through the process. The example below is a generated illustration, not a figure from a real project. Suppose you are building a dataset of customer support conversations, where each record pairs a customer message with a good agent reply. Watch how the count and the quality change as each step is applied.

The diagram shows only the count. The rewrite step is absent from it because it did not change the size of the dataset, even though it carried out one of the most important quality improvements in the plan. To judge a dataset properly, you need both measurements at every step.

Acquisition changes both the size and the quality of a dataset. Steps such as rewriting and relabeling can improve quality without changing the record count. In the support-conversation example, rewriting 1,000 records left the total at 4,200 while making those records better.

Synthesis adds data from existing material. A common pattern is to write a small number of templates by hand, then use an AI model to produce many instances from them. Synthesis is useful for increasing the size of a dataset, particularly where real examples are scarce.

Its limitation is that the output inherits the shape of its templates. If each template produces too many instances, the dataset can become repetitive. The suggested fix is to write more templates and generate fewer instances from each one.

Annotation adds the labels or other information that make data usable for training. Manual annotation is central to a high-quality dataset, and it is rarely a single pass. Once annotators start working, a team often discovers that some labels are not helpful. The response is to revise the annotation guidelines and relabel the affected records.

Some errors are factual rather than stylistic, and those may call for a second group of annotators who check the first group's work. The cost of these loops is part of the acquisition budget, not an unexpected extra.