Benchmarks · Science & research
LAB-Bench
also: Language Agent Biology Benchmark
Can a model do the practical work of biology research — finding facts in the literature, reading figures and tables, planning protocols, and reasoning over DNA and protein sequences?
FutureHouseReleased 14 July 2024Live
LAB-Bench asks a narrower and more practical question than most knowledge tests: not whether a model knows biology facts, but whether it can do the surrounding work of a research project — pulling the right claim out of a paper, reading a gel image or a data table, troubleshooting an experimental protocol, or reasoning correctly about a DNA or protein sequence. Built by FutureHouse, a nonprofit developing AI systems for scientific research, it packages 2,457 multiple-choice questions into eight categories covering literature search, database and supplementary-material retrieval, figure and table reading, protocol troubleshooting, and cloning-workflow planning.
At release, no model came close to a competent researcher. The strongest model tested, Claude 3.5 Sonnet, averaged around 45% across the full question set, against roughly 76% for PhD-level biologists answering the same questions — and performance varied sharply by category, from 90% on table-reading down to well under half on figure interpretation and cloning scenarios. The authors also showed the multiple-choice format could flatter models: when cloning questions were rewritten as open-response rather than pick-one-of-four, Claude 3.5 Sonnet’s score fell to around 20%, evidence that some of the multiple-choice accuracy came from eliminating implausible distractors rather than working the problem through.
LAB-Bench sits alongside a wider push, visible in later work such as OpenAI’s PaperBench and ChemBench, to test AI systems on the actual mechanics of research rather than on recall — the argument being that a model capable of accelerating science needs to clear these narrower, messier tasks, not just score well on exam-style questions. FutureHouse kept a fifth of each subtask private specifically to guard against the benchmark being absorbed into later training data, a precaution aimed at keeping the human–model gap it measured honest as models improved.
The set
2,457 multiple-choice questions across eight task categories (literature Q&A, database and supplementary-information retrieval, figure and table interpretation, protocol troubleshooting, sequence manipulation and cloning-workflow reasoning). About 80% of each subtask is public on Hugging Face; 20% is held back to monitor training contamination.
Example
A LitQA2 literature-recall question from the paper appendix: "What effect does expression of the ATPase-deficient E251Q mutant of the Spindle E protein in silkworm cells have on the levels of the mature piwiRNAs piR1712 and piR2986?" (Ideal answer: Decrease. Distractors: Increase, No effect.)arxiv.org
Where it stands
At release, the best model (Claude 3.5 Sonnet) scored around 45% against a roughly 76% human PhD-biologist baseline, and free-form (rather than multiple-choice) grading dropped scores further — the authors found models could exploit weak distractors, so a wide human/model gap likely remains.