AI2D

Long before today’s vision-language models existed, researchers at the Allen Institute for AI and the University of Washington noticed that a huge amount of human knowledge is encoded not in photographs but in diagrams — the water cycle, a food web, the parts of a flower — where arrows and labels carry as much meaning as the shapes they connect. Their 2016 dataset paired more than 5,000 such science-textbook diagrams with over 15,000 multiple-choice questions, annotated with a structural representation the authors called “Diagram Parse Graphs” that captured each diagram’s text, arrows and the relationships between them.

AI2D predates the current generation of general-purpose multimodal models by several years, but it survived the transition intact: because diagram reading is a distinct skill from photo understanding — closer to structured reasoning than to object recognition — it became a standard fixture in the evaluation suites of modern vision-language models rather than being replaced by a newer benchmark. Most current reports use a “with mask” variant that overlays the answer options directly onto the image, closing off any shortcut where a model reasons about the choices without properly reading the diagram.

By February 2025, models were clearing that bar comfortably: a comparison table in Alibaba’s Qwen2.5-VL technical report put InternVL2.5-78B highest at 89.1%, ahead of Gemini 1.5 Pro (88.4%), GPT-4o (84.6%) and Qwen2.5-VL-72B itself (83.5%). Scores clustering in the mid-to-high 80s across several unrelated labs suggest AI2D, like other diagram- and chart-reading benchmarks from its era, is approaching the point where it no longer discriminates cleanly between frontier systems.