Evaluating LLM performance across languages

Imagine a student who is most comfortable writing in Telugu. They ask a chatbot a detailed question about a school subject. The answer comes back weaker than it would have been in English: some sentences feel off, and a few facts are hard to trust. The student did nothing wrong, and the question was not harder than an English one. The gap comes from how the model was built and what it was taught on.

This article follows that gap through three places where it shows up. The first is the data a model learns from. The second is the translation shortcuts people use to work around the gap. The third is the way text is split into tokens, which affects both speed and price. For each one, we will trace what happens to the message step by step, so you can see where the quality or the cost is lost.

A model learns patterns from the text it is trained on. If a language appears only rarely in that text, the model has fewer examples to learn from, and it tends to perform worse in that language. Think of it as a learner who has read thousands of pages in one language and only a handful in another. Their answers in the less-read language will usually be shakier, even when they are equally clever.

The source material gives a concrete case. Telugu, Marathi, and Punjabi are the three languages where GPT-4 performed worst on the MMLU benchmarks. These same three languages are among the most under-represented in Common Crawl, a large web crawl dataset. The two facts line up: the languages with the least material in the training data are the ones where the benchmark results were weakest.

Low representation is not the whole story. The source notes that a language's structure and the culture it carries can make it harder for a model to learn, even when there is a reasonable amount of text. Coverage tells you how much material exists. Structure and culture affect how difficult that material is to learn from.

Because LLMs are generally good at translation, a tempting workaround is to translate every non-English query into English, let the model answer in English, and then translate that answer back into the original language. Many people use this approach. The source treats it as a reasonable but imperfect option, for two separate reasons.

The first reason is a dependency. The translation step only works well if the model understands the original language sufficiently. For an under-represented language, the translation itself can be the weak point, so the workaround quietly assumes a capability that may be missing. If the first step is shaky, every later step inherits the problem.

The second reason is information loss. Some languages mark the relationship between speakers directly in their pronouns. Vietnamese is the source's example. When such a sentence is translated into English, the various pronouns all become I and you, so the information about how the two speakers relate to each other disappears. The English model never sees it, and the translated-back reply cannot restore what was dropped.

Read the pipeline from left to right. The loss happens at the first translation step, and the model never gets a chance to recover it. The second translation step only changes the wording back into the original language; it cannot bring back a distinction that was removed earlier. This is why round-trip translation is a workaround rather than a complete fix.

Models do not read text as characters or words. They read tokens, and the same meaning can need a different number of tokens in different languages. This matters for speed and price. According to the source, a model's inference latency and cost are both proportional to the number of tokens in the input and the response. If a language needs more tokens to express the same meaning, every request in that language takes longer and costs more, even when the model's quality is the same.

The source gives a benchmark to make this concrete. Yennie Jun tested GPT-4 on MASSIVE, a dataset of one million short texts translated across 52 languages. For the same meaning, Burmese and Hindi needed far more tokens than English or Spanish. The median token length was 7 in English, 32 in Hindi, and 72 in Burmese.

Quality problems are not the only way language affects a model. The source reports that model behavior can also differ by language in ways that matter for safety. NewsGuard tested ChatGPT-3.5 in April 2023 by asking it to write misinformation articles about China. The same kind of request was made in English, in simplified Chinese, and in traditional Chinese. The model declined to produce false claims for six of seven English prompts, but it produced false claims all seven times in simplified Chinese and all seven times in traditional Chinese. The source states that the cause of this difference is unclear.

Cost and speed are a separate axis. Tokenization length, the translation step, and training data share each feed into what a user actually experiences. The concept map below shows how these factors connect to output quality and to the per-message cost.