MMBench
How consistently, not just how often, does a vision-language model get a multiple-choice visual question right — even when the answer options are shuffled?
OpenCompass / Shanghai AI Laboratory (Liu, Duan, Chen, Lin et al.)Released 12 July 2023
Multiple-choice questions have a well-known weakness: a model can get lucky, or learn to favour a particular answer position, without actually reasoning about the image. MMBench, built by the OpenCompass team at Shanghai AI Laboratory, was designed specifically to close that loophole. Its roughly 3,000 questions span 20 fine-grained visual abilities, from basic perception up to multi-step reasoning, and are asked in both English and Chinese — but its real innovation is CircularEval, a scoring protocol that re-asks each question several times with the answer options rotated into a new order and only credits a model that answers correctly under every rotation.
That design made MMBench harder to game than earlier multiple-choice vision-language benchmarks, and it was folded into OpenCompass’s broader evaluation toolkit, VLMEvalKit, which many open-weight model releases use to report standardised scores. The original 2023 paper did not headline a single accuracy figure; comparative numbers accumulated instead as later models reported against it. By February 2025, Alibaba’s Qwen2.5-VL-72B reported 88.4% on the “MMBench-V1.1-EN” test variant in its own technical report, ahead of GPT-4o’s 83.1% and Claude 3.5 Sonnet’s 80.9% on the same table.
MMBench remains a fixture of academic and open-weight vision-language papers rather than a headline figure in consumer model launches, and scores in the high 80s by 2025 suggest it is approaching the saturation earlier multiple-choice benchmarks such as MMLU reached. OpenCompass has continued to expand the MMBench family, including video and creative-writing variants, following the pattern of other benchmarks in this record that spawn harder successors once the original stops discriminating between top models.
The set
Roughly 3,000 multiple-choice questions covering 20 fine-grained ability dimensions, from basic object recognition up to multi-step reasoning, presented bilingually in English and Chinese. Its distinguishing feature is CircularEval: each question is re-asked several times with the answer choices rotated into a different order, and a model is only marked correct if it answers correctly on every rotation — a check against models that pattern-match a position rather than reason about the image.
Example
Which one is the correct caption of this image? A) A woman is riding a motorcycle down the street. B) The house appears to be clean and beautifully decorated. C) An elephant is chasing a dog around in the dirt. D) A woman is walking her dog and has stopped at a corner and is looking at all the lights and buildings. (Answer: A)huggingface.co
Where it stands
Widely used in academic and open-weight model releases via the OpenCompass evaluation toolkit; by February 2025 top models were reporting scores in the high 80s on the MMBench-EN v1.1 variant, with GPT-4o and Claude 3.5 Sonnet already in the low-to-mid 80s eight months earlier.
How the top score changed hands
Current best: Qwen2.5-VL-72B — 88.4% (MMBench-V1.1-EN test) Most recent score independently confirmed from an official technical report, ahead of GPT-4o (83.1%) and Claude 3.5 Sonnet (80.9%) on the same table; later 2025-26 frontier models plausibly score higher but are not confirmed here.