MMMU

MMMU asks whether a model can use an image, not just describe it. Each question comes from a real college exam, quiz or textbook and is genuinely unanswerable without reading the accompanying figure — a circuit diagram, a chemical structure, a music score, a map. Researchers at Ohio State University and the University of Waterloo built it that way deliberately, spreading 11.5K questions across six broad disciplines and 183 subfields so that no single visual skill could be gamed in isolation.

At launch in November 2023, the best available systems were well short of expert level: GPT-4V and Gemini Ultra scored 56% and 59% against a much higher human-expert baseline. Progress from there was rapid but shallow, and a year later the same team released MMMU-Pro to close the gap: it filters out any question a text-only model could already answer, expands the multiple-choice options, and adds a “vision-only” mode that renders the whole question as a single image, closing off the option of reading past the picture entirely. Reported scores dropped by 17 to 27 percentage points across the same models.

MMMU-Pro has since become one of the standard multimodal citations in frontier model announcements. Scores that sat around 51% for GPT-4o, Claude 3.5 Sonnet and Qwen2.5-VL-72B in February 2025 had climbed to the low 80s by the end of the year, with Gemini 3 Pro reporting 81.0% and the cheaper Gemini 3 Flash edging it out at 81.2% a few weeks later. Unlike some earlier multimodal benchmarks, it has not obviously saturated, and continues to sit alongside newer, harder successors such as CharXiv Reasoning in the eval suites frontier labs report at release.