MVBench
also: MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
Whether a multimodal model can answer a question about a video that requires genuine temporal reasoning — motion, order, counting, causality — rather than being answerable from a single freeze-framed image.
OpenGVLab, Shanghai AI Laboratory, with Nanjing University, Fudan and University of Hong KongReleased 28 November 2023Live
Most video benchmarks for multimodal models can be gamed by looking at a single frame. MVBench was built to close that loophole: researchers from Shanghai AI Laboratory’s OpenGVLab, together with Nanjing University, Fudan and the University of Hong Kong, took existing video annotations — action recognition labels, object-tracking traces and the like — and converted them into around 4,000 multiple-choice questions across 20 task types, each one deliberately unanswerable from a still image. Answering correctly requires tracking motion, ordering events, counting occurrences or reasoning about cause and effect across a clip.
At launch in November 2023, the gap between purpose-built video models and general multimodal models was stark: the paper’s own VideoChat2 baseline scored 51.1% against GPT-4V’s 43.5%, more than 15 points ahead of the field. That gap narrowed quickly. By early 2025, Alibaba’s Qwen team reported Qwen2.5-VL-72B reaching 70.4% in a comparison table that also placed Gemini 1.5 Pro at 60.5% and GPT-4o at 64.6% — a roughly 20-point rise across the field in little more than a year.
MVBench has no official successor the way some saturated benchmarks do, and it remains a routine entry in multimodal model technical reports. Its scores are not directly comparable to single-image perception benchmarks, since the whole point of the design is to penalise models that treat video as a slideshow rather than a sequence.
The set
Roughly 4,000 multiple-choice questions across 20 temporal-reasoning task types (around 200 per task), built with a 'static-to-dynamic' method that converts existing video annotations — action recognition, object tracking and similar labelled data — into QA pairs. Scored on plain multiple-choice accuracy, chosen specifically to avoid the scoring bias of open-ended, LLM-judged answers.
Example
An action-antonym item, verbatim: 'What is the action performed by the person in the video?' with options including 'Scattering something down' and the correct answer, 'Piling something up' — deliberately similar-sounding options that punish guessing from a single freeze-frame.huggingface.co
Where it stands
Scores have climbed well past the launch baselines and the benchmark remains a standard citation in multimodal technical reports into 2025, though it has not been formally succeeded the way some earlier benchmarks have.
How the top score changed hands
- November 2023VideoChat2 (paper's own baseline)51.1%Surpassed GPT-4V (43.5%) and prior video MLLMs by more than 15 points at launch.
- February 2025Qwen2.5-VL-72B70.4%Reported in a comparison table alongside Gemini 1.5 Pro (60.5%) and GPT-4o (64.6%).
Current best: Qwen2.5-VL-72B — 70.4% Highest score in a third-party comparison table that also lists Gemini 1.5 Pro (60.5%) and GPT-4o (64.6%); not from MVBench's own leaderboard, and later frontier models may score higher without a comparable independently-checked figure here.