Rule-Based Data Synthesis

Imagine you need thousands of realistic records to test or train a system, but you cannot use real customer records to do it. One practical answer is to stop thinking about individual records and instead design the shape a record must have. Once that shape is fixed, you can produce as many instances of it as you like. This article follows that idea from the simplest case, a fill-in-the-blank form, through to the way transformed copies of existing data can expand a dataset.

The source gives a concrete case: to create a credit card transaction, you start with a transaction template and use a random generator, such as Faker, to populate each field in that template. The template is the fixed part. The random generator is the source of variation. Each field in the template is a slot, and the generator decides what goes into each slot.

Work through a small case. The transaction template has three slots: a transaction identifier, a date, and a time. Before any generation happens, the template is only a description of what a transaction must look like. Nothing in it is a real transaction yet. The template also states the expected format of each slot, so the generator must produce values in those formats.

Notice the order of events in this pipeline. The structure of the output is decided by the template before generation starts. The variety of the output comes from the generator, one field at a time. This split is what makes the method predictable in shape and varied in content.

A random generator fills each template field, so the structure stays fixed while the values vary. You can think of the template as a contract about layout and format, and the generator as the component that produces many different acceptable values under that contract. Changing the generator changes the values; changing the template changes the shape of every record produced from it.

Template-based generation separates structure from variety. The template fixes the fields and their expected formats, and the random generator supplies the values. Because the structure is decided first, every output has the same fields in the same order, even though no two outputs need to contain the same values.

Now run the same template twice. The field names stay exactly the same in both records, while the values assigned to those fields differ. The map below shows which value each field receives in each generated record.

Each column in the map has the same keys, which is the structure the template guarantees. The values in each column come from the generator and are different from one another. A learner reading only one record cannot tell which values were drawn from the generator and which were fixed by the template, which is why it helps to compare several records side by side.

Templates are not limited to records with named fields. The source lists two other broad uses. The subsections below describe each one.

Templates create data from nothing but a structure. Data augmentation works differently: it starts with data you already have and produces new data by applying simple transformations to it. The source describes this as procedurally generating new data from existing data.

For images, the transformations named in the source are random rotation, cropping, scaling, and erasing part of the image. Each transformed copy is a new training example that is derived from an original one. The diagram below shows how one original image becomes several new examples.

The reason this works is that these transformations are expected to preserve meaning. The source gives two examples: a flipped image of a cat should still be a cat, and a slightly cropped image of a soccer game should still be a soccer game. The label stays the same while the pixels change.

The main practical motivation appears in the source's discussion of sensitive data. Transaction data is sensitive, so many fraud detection models are first trained on synthetic transaction data generated from templates like the one above. The purpose is to show that the model is feasible before it is given access to real data.

Think of the project timeline in three phases. First, the team designs the template and generates synthetic records, so the pipeline and model can be built and tested. Second, the model is shown to work on data that has the right shape. Third, when access to real data is permitted, the same model is run on that data. Synthetic data fills the gap between the start of the project and the arrival of real data.

The source also points to a large-scale use of this idea. DeepMind trained an Olympiad-level geometry model, AlphaGeometry, using 100 million synthetic examples. That scale shows why generating data from structured templates can be worth the effort when real examples are limited.

The two kinds of data are easy to confuse, because both have the same field names. The table sets them side by side, using what the source describes about each.