Curating Data for Specific Languages and Domains

Suppose you are building a language model and you need a huge amount of text to train it. The easiest move is to take whatever is already available and feed all of it in. Nothing has to be negotiated, licensed, or written by hand, and the volume looks impressive. This shortcut is the starting point for the problems in this article. Collecting data is easy. Deciding which data is worth keeping is the hard part, and that choice shapes what the finished model can and cannot do.

A widely used starting point for web-scale training text is Common Crawl. It is produced by a nonprofit organization that crawls websites across the internet on an irregular schedule rather than continuously. In 2022 and 2023, this organization collected roughly 2 to 3 billion web pages each month. Because the collection is so large and so freely available, it became a convenient raw material for many projects.

Google offers a cleaned version of this material called the Colossal Clean Crawled Corpus, usually shortened to C4. Cleaning helps, but the source notes that the quality of C4 is questionable to a certain extent. This tells you something important: a cleaned subset still inherits many of the problems of the collection it came from. Cleaning is not the same as curating for a purpose.

Bulk web text is not sorted by usefulness. A crawl captures whatever pages it reaches, and the open web includes clickbait, misinformation, propaganda, conspiracy theories, hateful content, and sites that most careful readers would avoid. These quality problems are the central reason curation is needed at all. A dataset does not become trustworthy simply because it is large.

The problem can be checked against outside ratings. A study by the Washington Post examined the one thousand most common websites in the dataset and found that several media outlets among them rank low on NewsGuard's scale for trustworthiness. In plain terms, a widely used source of training text contains a meaningful amount of fake news, and this is true even of its most frequently appearing sites.

If the problems are so visible, why does problematic data keep appearing in training pipelines? Part of the answer is that the same convenient source is reused. Variations of Common Crawl are used in most foundation models that disclose their training data sources, including OpenAI's GPT-3 and Google's Gemini. The source also points to a less visible effect: many companies have stopped disclosing their training data sources, partly to avoid scrutiny from the public and from competitors. The full picture of which sources were used is therefore often hidden from outside observers.

Some teams respond to quality problems with heuristics, which are simple rules that screen data before it reaches training. One documented example comes from OpenAI, which used only the Reddit links that received at least three upvotes to train GPT-2. The idea is that a link other people chose to endorse is more likely to be worth keeping than one nobody reacted to. Follow a single web document through this rule. It enters the filter, and the first question is whether it is a Reddit link. If it is, the filter counts its upvotes. A link with three or more upvotes is kept. A link with fewer is removed.

The source is candid about this trade-off. The rule does help screen out links that nobody cares about, but Reddit is not exactly a model of propriety and good taste. An upvote count measures popularity, and popularity does not guarantee accuracy or appropriateness. A heuristic removes some noise while leaving other problems in place. In the worked example, the filter could not tell a well-endorsed true claim from a well-endorsed false one, because it never looked at the content.

The remedy the source proposes is to curate datasets that align with your specific needs. The source describes a direction that focuses on specific languages and domains, aiming for a foundation that is broad enough to be useful but specialized enough to serve applications in those areas. The contrast is clearest when you set a collect-everything approach beside a curate-to-needs approach.

According to the source, specialized models can be built in two ways. A language- or domain-specific foundation model can be trained from scratch, or a general-purpose model can be finetuned further on specialized material. Either route depends on having data that actually reflects the target. Curation is what makes that data available.

A natural objection is to train on everything, general and specialized data together, so the model can do everything. The source acknowledges that many people do this. It also warns that training on more data often requires more compute resources and does not always lead to better performance. More volume is not a free improvement.

The strongest argument for curation is that quality can matter more than quantity. The source states that a model trained with a smaller amount of high-quality data might outperform a model trained with a large amount of low-quality data. This directly challenges the assumption that more text always produces a better model.

The source gives a concrete case. Gunasekar et al. (2023) used 7B tokens of high-quality coding data to train a 1.3B-parameter model. That model outperforms much larger models on several important coding benchmarks. Trace the two training sets side by side. One is small and carefully chosen. The other is large and unfiltered. The diagram lays out the contrast the source is making.