Benchmarks · Language & multilingual
Global-MMLU
also: Global MMLU
Whether a model's MMLU-style general knowledge holds up across 42 languages, and separately, whether its score depends on knowledge specific to a particular culture rather than being culturally neutral.
Cohere For AIReleased 4 December 2024Live
Global-MMLU is a response to a problem the original MMLU translation efforts largely ignored: a knowledge question does not travel cleanly across languages just because its words do. Cohere For AI and a wide set of academic collaborators rebuilt the MMLU test set across 42 languages using a mix of machine translation, professional human translation and community post-editing, then went a step further and had annotators mark every question as either culturally sensitive — requiring knowledge specific to a particular region — or culturally agnostic.
That labelling was the point of the exercise. The paper found a substantial share of MMLU’s questions leaned on culturally specific knowledge, and that questions with a geographic reference skewed heavily toward North America and Europe. Evaluating 14 models across nine model families, the authors found scores were markedly less stable on low-resource languages, and — more pointedly — that a model’s ranking against its competitors could shift depending on whether it was scored on the full question set or only the culturally-sensitive subset.
Global-MMLU sits alongside OpenAI’s MMMLU as one of two major multilingual descendants of MMLU, distinguished mainly by its broader language coverage and its explicit accounting for cultural bias rather than treating translation as a solved, neutral step. It also ships a smaller “lite” 23-language subset intended to make routine multilingual evaluation cheaper to run.
The set
MMLU questions covering all 42 languages combine machine translation, professional human translation and community post-editing rather than a single translation pass; every question is additionally labelled as culturally sensitive (dependent on region-specific knowledge) or culturally agnostic, with 2,850 annotated questions per language and a smaller 'lite' subset of 6,000 questions across 23 languages for cheaper evaluation runs.
Example
'Find the degree for the given field extension Q(sqrt(2), sqrt(3), sqrt(18)) over Q' — an MMLU abstract-algebra question translated into Amharic, with four numeric options (0, 4, 2, 6); the correct answer is 4.huggingface.co
Where it stands
Positioned against OpenAI's MMMLU as a broader and more rigorously translated alternative; the paper reports model rankings shift depending on whether scoring uses the full set or only the culturally-sensitive subset.