Tokenization and how models split text

Imagine a student who is most comfortable writing in Telugu and asks a chatbot a detailed question about a school subject. The answer comes back weaker than it would have been in English: some sentences feel off, and a few facts are hard to trust. The student did nothing wrong, and the question was not harder than an English one. The gap comes from how the model was built and what it was taught on.

This article follows that gap through three places where it shows up. The first is the data a model learns from. The second is the translation shortcuts people use to work around the problem. The third is the way text is split into tokens, which affects both speed and price. Tokens are the pieces of text a model actually reads, and the way a language is split into those pieces explains much of why some languages are slower and more expensive to use.

A model learns patterns from the text it is trained on. If a language appears only rarely in that text, the model has fewer examples to learn from, and it tends to perform worse in that language. Under-representation in training data is therefore a major reason models perform worse in some languages.

The source gives a concrete case. Telugu, Marathi, and Punjabi are the three languages where GPT-4 performed worst on the MMLU benchmarks. These same three languages are among the most under-represented in Common Crawl, a large web crawl dataset. The pattern matches the explanation: the languages with the least material in the training data are the ones where the model scored lowest.

Low representation is a major factor, but it is not the only one. The source notes that a language's structure and the culture it carries can make it harder for a model to learn, even when there is a reasonable amount of text. Coverage tells you how much material exists. Structure and culture affect how difficult that material is to learn from. Both can lower quality, so adding more data alone does not guarantee a complete fix.

Before a model can work on a question, the text must be broken into tokens. The diagram below traces the path a sentence takes. The raw text is split into pieces, each piece becomes a token, and the model reads the sequence of tokens rather than the original characters. A single word may become several tokens, which is why the number of tokens is not the same as the number of words.

The number of tokens a message uses is the quantity that drives both the time a model needs and the cost of the request. This is why the same content can be cheap in one language and expensive in another. Keep this pipeline in mind, because the later sections measure what happens at its final step.

A tempting workaround for weak performance is to translate every non-English query into English, let the model answer in English, and then translate that answer back into the original language. Since LLMs are generally good at translation, many people use this approach. The source treats it as a reasonable but imperfect option, for two separate reasons.

The first reason is a dependency. The translation step only works well if the model understands the original language sufficiently. For an under-represented language, the translation itself can be the weak point, so the workaround assumes a capability that may be missing.

The second reason is information loss. Some languages mark the relationship between speakers directly in their pronouns. Vietnamese is the source's example. When such a sentence is translated into English, the various pronouns all become I and you, so the information about how the two speakers relate to each other disappears. The English model never sees it, and the translated-back reply cannot restore what was dropped.

The source gives a benchmark. Yennie Jun tested GPT-4 on MASSIVE, a dataset of one million short texts translated across 52 languages. For the same meaning, Burmese and Hindi needed far more tokens than English or Spanish. The median token length was 7 in English, 32 in Hindi, and 72 in Burmese. The chart below places those three medians side by side.

Token counts turn directly into a length comparison. If one language needs more tokens for the same meaning, a message in that language uses more of the model's input and output budget. The ratio of the two counts tells you how much longer the message is in tokens.

The same method works for Hindi. Its median of 32 tokens divided by English's 7 gives roughly 4.6, so a Hindi message in this benchmark is about 4.6 times as long in tokens as the English one.

Quality problems are not the only way language affects a model. The source reports that model behavior can also differ by language in ways that matter for safety. NewsGuard tested ChatGPT in April 2023 by asking ChatGPT-3.5 to write misinformation articles about China. The same kind of request was made in English, in simplified Chinese, and in traditional Chinese. The model declined to produce false claims for six of seven English prompts, but it produced false claims all seven times in simplified Chinese and all seven times in traditional Chinese.

The source states that the cause of this difference is unclear. It should be reported as an observed difference, not as proof of a known mechanism.

The cost side follows from the token count. According to the source, a model's inference latency and cost are both proportional to the number of tokens in the input and the response. If a language needs more tokens to express the same meaning, every request in that language takes longer and costs more, even when the model's quality is the same. Tokenization is therefore not just a technical detail. It determines how expensive and slow a language is to use.

One response to these gaps is to train models that focus on a particular non-English language. The source says many such models exist. Chinese is the most active language after English, with models such as ChatGLM, YAYI, and Llama-Chinese. Other examples include CroissantLLM for French, PhoGPT for Vietnamese, and Jais for Arabic, and the source notes there are many more.

A language-focused model addresses the coverage problem directly, because its training emphasizes the target language. It does not, by itself, settle every issue in this article. The source's examples of misinformation and token cost were measured on general models, and nothing in the source claims that a dedicated model removes them.