Privacy and Safety in Training Data

A model learns whatever patterns are present in the examples it receives. If those examples contain stray markup, content the model should never use, or records that are simply poor, the model absorbs those problems along with everything useful. This is the reason the people preparing data must decide, before training starts, what stays, what goes, and what might be worth keeping in a smaller form.

Many public datasets are assembled by collecting text from across the internet. Because web pages are written in markup languages, text pulled from them often carries HTML and Markdown markup inside the text itself. The source describes these formatting tokens as extraneous. Unless the goal is to teach a model to read those tags, they are noise, and removing them is the first cleaning step.

The source summary makes a practical claim about this step: removing formatting tokens can improve accuracy while also cutting the length of the input. Both effects come from the same action. Fewer meaningless characters means less text for the model to process, and the remaining text is closer to the content the project actually cares about.

Cleaning also means enforcing rules. Data has to be checked against the policies that govern the project, and anything that does not comply must be removed. The source names several categories that fall under this heading: personally identifiable information (PII), sensitive data, copyrighted material, and content considered toxic. The techniques for handling those categories are covered in an earlier chapter on privacy and safety, so this article focuses on the filtering step itself.

Not every violation requires discarding a whole record. A dataset may contain columns that the project is not permitted to use, such as a zip code, a name, or a gender value. Those fields should be dropped before the model sees the data, even when the rest of the record is perfectly usable.

After compliance cleaning, the data may still contain records that are poor in quality even though they are allowed and properly formatted. The source describes a method for finding them that combines two activities. The first is looking closely at the data by hand. The second is turning what was noticed into rules that can be applied across the whole collection.

Manual inspection reveals patterns. When a person repeatedly sees the same kind of weak record, that pattern can become a heuristic, which is a rule that flags similar records automatically. The source also notes that some heuristics depend on context, such as the position of a record within an annotation session. A rule like this cannot be read off the record alone, which is one reason heuristics need to be checked against what a person actually observed.

The steps described so far are not independent choices. They form a sequence in which each stage hands a smaller and cleaner collection to the next. Follow one raw record through that sequence. First its formatting tokens are stripped. Then it is checked against project policies, and any non-compliant record or field is removed. Next manual inspection and heuristics look for low-quality patterns. Finally, if the project has more data than it can use, an optional pruning stage selects a subset before training.

Notice where records leave the pipeline. Non-compliant records and low-quality records are dropped at different points, and the optional pruning stage only applies when the collection is larger than the project can use. Each stage works on what the previous stage left behind, so the training set is the end result of every filter, not any single one.

Sometimes the problem is not bad data but too much of it. The source notes that if there is more data than you need or can afford to use, for example because of your compute budget, you can filter further. Two techniques are named. Active learning selects the examples that are most helpful for the model to learn from. Importance sampling selects the examples that are most important to the task. Both are ways of keeping a subset chosen on purpose, rather than a random or arbitrary portion.

The contrast in the diagram is the core idea. A random or arbitrary portion keeps whatever happens to be included, with no judgment about value. Active learning and importance sampling instead judge each example and keep the ones that help most. The result is a smaller dataset whose contents were decided deliberately.

Both active learning and importance sampling depend on a single requirement, which the source states directly: their efficiency depends on whether you have a good way to evaluate the importance of each training example. Choosing examples is only as good as the measure used to judge them. A weak measure keeps the wrong examples, while a strong measure lets a much smaller set carry most of the value.

The source ties this to cost. Meta researchers, in their study of data pruning (Sorscher and colleagues, 2022), concluded that discovering good data-pruning metrics can significantly reduce the resource costs of modern deep learning. The practical message is that a good metric is an investment: it determines how much of a large dataset can be dropped without losing the examples that matter.