Benchmarks · Language & multilingual
Belebele
Whether a model can read a short passage and answer a factual question about it correctly, tested in parallel across 122 languages and dialects using exactly the same underlying questions.
Meta AIReleased 31 August 2023Live
Belebele tests the most basic form of language understanding — read a short passage, answer a factual question about it — but does so in parallel across 122 languages and dialects using exactly the same passages and questions throughout. Meta AI built it from 488 passages drawn from its own FLORES-200 translation set, with 900 four-option multiple-choice questions per language, so that a model’s score in Icelandic or Amharic is directly comparable to its score in English rather than an approximation across differently-constructed tests.
The original evaluation produced a counterintuitive result: purpose-built multilingual encoder models, trained specifically for cross-lingual understanding, outperformed the large generative language models of the day. InfoXLM reached 60.0% average accuracy in a translate-train-all setting, ahead of GPT-3.5-turbo’s 51.1% zero-shot average across all 122 languages — a reminder that scale and general capability did not automatically translate into breadth across low-resource languages the way English-centric benchmarks might suggest.
Because it is cheap to run, fully parallel and covers far more languages than most alternatives, Belebele became a standard first check on a model’s basic multilingual comprehension, often reported alongside deeper reasoning benchmarks like MGSM rather than in place of them. Its reliance on FLORES-200 passages also ties its usefulness to that dataset’s own coverage and quality.
The set
900 multiple-choice reading-comprehension questions per language, built from 488 passages drawn from the FLORES-200 translation set, each with four answer options; because every language uses the same underlying passages and questions, scores are directly comparable across languages rather than approximated.
Example
A passage on accordion-playing technique is followed by: 'According to the passage, what would not be considered an accurate tip for successfully playing the accordion?' The correct option flags 'hit the keys with more force for extra volume' as wrong — the passage credits bellows pressure instead.huggingface.co
Where it stands
Widely used as a cheap, parallel check on basic multilingual comprehension; early large language models trailed purpose-built multilingual encoder models on it despite their broader capabilities.