Data Pruning and Compute Budgets

A model learns whatever patterns are present in the examples it receives. If those examples contain stray markup, content the model should never use, or records that are simply poor, the model absorbs those problems along with everything useful. This is the reason data preparation is a decision-making step, not a formality. Before training begins, the people preparing the data must decide what stays, what goes, and what might be worth keeping in a smaller form.

This article follows that decision-making from start to finish. It covers three jobs: cleaning data that is not usable as it stands, detecting records that are low in quality even though they are allowed, and filtering a collection that is larger than the project can afford to use. The stages are not independent choices. Each one hands a smaller and cleaner collection to the next.

Many public datasets are assembled by collecting text from across the internet. Web pages are written in markup languages, so text pulled from them often arrives wrapped in HTML tags. These tags can be angle-bracketed page structure, styling hints, or links. Unless the goal is to teach a model to read those tags, they are noise. The first cleaning task is therefore to remove extraneous formatting tokens, including HTML and Markdown markup that ended up inside the text itself.

The source states that removing these formatting tokens can improve accuracy while cutting input length. The diagram below traces what happens to one scraped fragment. The markup tags are the parts that change; the readable text stays.

Cleaning also means enforcing rules. Data has to be checked against the policies that govern the project, and anything that does not comply has to be removed. The source names several categories that fall under this heading: personally identifiable information (PII), sensitive data, copyrighted material, and content considered toxic. Techniques for handling these categories are covered in an earlier chapter on privacy and safety, so this article focuses on the filtering step itself.

Non-compliant material is not always a whole record. Sometimes it is a set of fields. A dataset may contain columns the project is not permitted to use, such as a zip code, a name, or a gender value. Those fields should be dropped before the model sees the data, even when the rest of the record is perfectly usable. The source's guidance is that removing a field is often better than removing the whole record, because the usable part of the record is preserved.

After compliance cleaning, the data may still contain records that are poor in quality even though they are allowed and properly formatted. The source points to a method for finding them that combines two activities. The first is looking closely at the data by hand. The second is turning what is noticed into rules that can be applied across the whole collection. The two approaches work together: manual inspection reveals patterns, and those patterns become heuristics.

The steps described so far form a sequence. Formatting tokens are removed first, compliance rules are enforced and non-compliant fields are dropped, and heuristics and manual findings remove low-quality records. Only then does the collection reach an optional filtering stage. That stage is for projects that have more data than they can use. The diagram shows the order the source implies, with each stage passing a smaller and cleaner collection to the next.

Sometimes the problem is not bad data but too much of it. The source notes that if there is more data than you need or can afford to use, for example because of your compute budget, you can filter further. Two techniques are named. Active learning selects the examples that are most helpful for the model to learn from. Importance sampling selects the examples that are most important to the task. Both are ways of keeping a subset chosen on purpose, rather than a random or arbitrary portion.

The diagram shows the general shape of this selection. Every example in the pool is judged, the examples that score as most helpful or most important are kept, and the rest are set aside. The result is a smaller collection that still reflects the choices made about what matters. The method is only as sound as the judgement behind the scores, which is the subject of the next section.

Both active learning and importance sampling depend on a single requirement, which the source states directly: their efficiency depends on whether you have a good way to evaluate the importance of each training example. Choosing examples is only as good as the measure used to judge them. A weak measure keeps the wrong examples, and a strong measure lets a much smaller set carry most of the value.

The reason this matters is that a pruning score works like a cutoff. Records scoring above the line are kept and records below it are removed. If the score is accurate, the good examples land above the line and the poor ones below it. If the score is noisy, some valuable examples fall below the cutoff and are discarded, while some poor examples rise above it and are kept. In both cases the cutoff itself does not change; only the quality of the score decides which records end up on each side.

The source ties the value of this work to cost. Meta researchers, in their study of data pruning (Sorscher and colleagues, 2022), concluded that discovering good data-pruning metrics can significantly reduce the resource costs of modern deep learning. The practical message is that a good metric is an investment, because it determines how much of a large dataset can be dropped without losing the examples that matter.