Benchmarks · Language & multilingual
MMMLU
also: Multilingual Massive Multitask Language Understanding
Whether a model retains its general-knowledge accuracy when the same MMLU questions are asked in a language other than English.
OpenAIReleased 23 September 2024Live
MMMLU takes the question format that made MMLU the default general-knowledge yardstick and asks whether a model’s score holds up outside English. OpenAI published the dataset on Hugging Face in September 2024: the same 57-subject, four-option multiple-choice test set, professionally translated by human linguists — not machine translation — into 14 languages spanning high-resource languages like German and Japanese and lower-resource ones like Swahili and Yoruba.
Because MMLU itself had become the standard single number attached to a model release, giving it a faithful multilingual counterpart let labs report a comparable figure for non-English performance rather than relying on ad hoc or machine-translated tests of uneven quality. OpenAI and most other frontier labs have since included an average MMMLU score, or scores broken out by language, in model system cards and release material.
MMMLU is a straight translation of MMLU’s content, which is also its main limitation: a question written for an American test-taker does not automatically test locally relevant knowledge just because it has been translated. Cohere For AI’s community-built Global-MMLU addressed that directly by annotating which questions depend on culturally specific knowledge, and OpenAI’s own later benchmark, IndQA, was framed explicitly as a response to the same gap for Indian languages.
The set
The MMLU test set, professionally translated by human linguists (not machine translation) into 14 languages — Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Brazilian Portuguese, Swahili, Yoruba and Simplified Chinese — preserving the original 57-subject, four-option multiple-choice format.
Example
'Find the degree for the given field extension Q(sqrt(2), sqrt(3), sqrt(18)) over Q' — the same abstract-algebra MMLU question professionally translated into Arabic (and the 13 other MMMLU languages), with four numeric options; the correct answer is 4.huggingface.co
Where it stands
OpenAI and most other labs report at least an average MMMLU number in model cards, but a community-built alternative, Global-MMLU, has broader language coverage and adds annotations for cultural bias that MMMLU lacks.