User Feedback Systems and Data Flywheels

Many people who start building a model expect the hard part to be the model itself. In practice, a team often spends more effort deciding what data the model should learn from. A dataset is rarely something you find ready-made. It usually has to be assembled from several places, cleaned, extended, and labeled before a model can use it. This assembly work is called data acquisition, and it is planned the way an engineer plans a supply chain: you need to know which inputs you require, where they can come from, what they cost, and what state they will be in when they arrive.

The goal is a dataset that is large enough, varied enough, and accurate enough for the task, collected in a way that respects user privacy and the rules that apply to the data. Those constraints shape every later choice. Keep one question in mind throughout this article: for each record in the eventual dataset, where did it come from, and what has happened to it since?

Data acquisition draws on four broad routes. Each one answers a different question about where a record comes from. The routes are not alternatives that you choose between once. A single dataset can contain records from several of them, and it can pass through annotation and synthesis more than once.

The four routes divide the work of acquisition into distinct kinds of effort. Each one has a different source of data, a different cost pattern, and a different role within a project. The subsections below take them in turn.

A dataset changes at every step of acquisition, and the clearest way to see how is to follow one through the process. This illustration is generated for teaching rather than drawn from a real project. Suppose you are building a dataset of customer support conversations, where each record pairs a customer message with a good agent reply. Each step can add records, remove them, improve them, or attach information to them. Watch how the count and the quality change as each step is applied.

Public data has a requirement that is easy to forget: a public dataset is not automatically safe to use. Before using one, check its license, and try to understand where its contents came from. A license that permits commercial use can still cover a dataset that includes material collected under different terms. Public data is worth using, but it must be inspected and validated like any other input.

The order in this pipeline matters. Searching comes first because a ready-made match can save effort. Validation comes before inclusion, so that no unchecked record reaches the training data. When a dataset fails, the gap is filled through purchasing, annotation, or synthesis rather than by accepting the dataset as it is.

Of the four routes, the one that usually matters most is the data your own application produces. The reason is distribution. Your application's records reflect the kinds of inputs your users actually send, in the proportions in which they actually send them. Public and purchased datasets were gathered for someone else's purpose, so their match to your task is often only partial. Matching the distribution of the data you care about is difficult to achieve with other sources, which is why application data is described as ideal for the task.

Application data takes three broad forms: user content that users create, system-generated data that records how the product is used, and user feedback. Each form can supply records for the task. The design of a feedback system is a topic of its own and is covered later in the course.

Application data can also feed back into the product. The useful pattern is a loop in which users generate data, that data is used to improve the product, and the improved product generates more useful data. Teams often call this a data flywheel. Each pass around the loop starts from a richer base than the last, so the next pass produces data that is more valuable than the one before.

Every turn of the loop adds data that is matched to real use, which makes the next round of improvement easier to target. A product that is used more generates more feedback, and that feedback is exactly the kind of data the task needs.