Multilingual Models and Language Representation

Large language models learn from enormous collections of text, and a large share of that text comes from the public web. Common Crawl is a widely used web dataset of this kind, and it is one of the datasets used for training LLMs. Before deciding what a model should learn from, it helps to know which languages actually fill the dataset. The answer is far from balanced: a small number of languages supply most of the text, and many others supply very little.

This article works through one practical question: when you open a language distribution table for Common Crawl, what does it tell you, and what does it leave out? You will read the table, turn two of its values into a single multiplier, and apply a cutoff that decides which languages appear at all. Along the way you will see why the term low-resource language has a specific meaning in this context.

The language table lists only languages that make up at least 1% of Common Crawl. Each row gives a language name, its language code, a population figure in millions, and the language's share of the Common Crawl data as a percentage. Keep the two numeric columns separate in your head. The population column measures how many people speak a language. The share column measures how much of this web dataset is written in that language.

The rows are sorted by the Common Crawl share column, so the first row is the largest contributor. The population column does not set the order. Chinese has a much larger population figure than German, yet German appears above Chinese in the share ordering. If you sorted by population instead, the list would look different, and it would answer a different question.

The source makes its central point with a comparison between the top two rows. English accounts for almost half of the data, at 45.88%, and the source describes this as eight times more prevalent than Russian, the second-most common language at 5.97%. The comparison turns two raw percentages into one multiplier that is easy to remember.

The same kind of reading applies further down the table. Dutch at 2.0585% and Vietnamese at 1.0299% are both listed, and their shares differ by about a factor of two. Dividing 2.0585 by 1.0299 gives roughly 2.0. The table supports comparisons like this one, but it says nothing about languages that are not in the table at all.

The table is built with a cutoff. Only languages with at least 1% representation in Common Crawl are listed. A language that does not meet the cutoff has no row at all. A reader of the table can only infer that such a language exists from the cutoff itself.

A helpful way to hold the definition is to ask two questions in order. First, is the language in the at-least-1% table for this dataset? If it is, it is listed. If it is not, it is considered low-resource in the sense the source uses. Asking in this order avoids a common confusion, which is trying to decide low-resource status from the population figure alone.

Follow two languages through the check. The first is Vietnamese, the smallest row in the table. The second is a hypothetical language, which is not in the source, and is used here only to show the unlisted path.

The table cannot tell you how much text sits in the unlisted group, only that each language in that group falls outside the list. Adding up the twelve listed percentages gives about 85.06%. That leaves roughly 14.94% of the dataset spread across languages outside the list. This remainder is a calculation from the table values, not a figure the source states directly.