DocVQA
Can a model locate and correctly read the specific piece of text in a scanned business or government document — a form, letter, report or table — needed to answer a natural-language question about it?
IIIT Hyderabad, CVC (Universitat Autònoma de Barcelona) & AmazonReleased 1 July 2020Saturated
Most document-understanding tasks before 2020 were built around clean, digitally-generated PDFs. DocVQA, published by researchers at IIIT Hyderabad and CVC in Barcelona with support from Amazon, instead used real scanned documents — forms, letters, memos, reports — the messy paper trail that businesses and governments actually produce. Its 50,000 questions over more than 12,000 images ask a model to find and read the specific value a question is after, whether that means a date on a letterhead, a figure buried in a table, or a handwritten annotation, and score answers with ANLS, a similarity metric that tolerates small OCR slips rather than demanding a perfect match.
The authors measured human accuracy at 94.36%, and for several years that figure stood as a meaningful ceiling: early document-AI systems, built from separate OCR and reasoning stages, fell well short of it. That changed as vision-language models learned to read text directly from image pixels rather than relying on a separate OCR pipeline bolted onto a language model. By February 2025, a comparison table in Alibaba’s Qwen2.5-VL technical report showed the field clustering at or above the human baseline: Qwen2.5-VL-72B led at 96.4%, with Claude 3.5 Sonnet, InternVL2.5-78B, Gemini 1.5 Pro and GPT-4o all in the 91–96% range on the same table.
With frontier models now matching or beating the original human ceiling, DocVQA functions less as a discriminator between top labs and more as a baseline competence check — a benchmark models are expected to pass rather than one that reveals meaningful differences between them. It remains, even so, a routine citation in vision-language model release material, and its scanned-document format anticipated the more demanding OCR and layout benchmarks, such as OmniDocBench, that later took over the role of separating frontier systems on document understanding.
The set
50,000 questions over more than 12,000 real scanned document images (forms, letters, reports, memos), covering tables, figures and free-form text. Scored with ANLS (Average Normalised Levenshtein Similarity), which gives partial credit for near-miss OCR errors instead of demanding an exact string match.
Example
Q: Mention the ZIP code written? A: 80202arxiv.org
Where it stands
Human accuracy was measured at 94.36% when the benchmark launched; by February 2025, several frontier vision-language models were matching or exceeding that figure, with Qwen2.5-VL-72B reporting the highest score in one widely-cited comparison table.
How the top score changed hands
- July 2020Human baseline94.36%Measured by the original authors as the accuracy ceiling for the benchmark at launch.
- February 2025Claude 3.5 Sonnet / Gemini 1.5 Pro / GPT-4o / InternVL2.5-78B / Qwen2.5-VL-72B95.2% / 93.1% / 91.1% / 95.1% / 96.4% (test, ANLS)From Qwen2.5-VL's own technical report; all five models cluster near or above the original 94.36% human baseline.
Current best: Qwen2.5-VL-72B — 96.4% (test, ANLS) Highest score in Qwen2.5-VL's own technical report comparison table, ahead of Claude 3.5 Sonnet (95.2%), InternVL2.5-78B (95.1%), Gemini 1.5 Pro (93.1%) and GPT-4o (91.1%) on the same table; later 2025-26 frontier models plausibly score similarly high but are not confirmed here.