Benchmarks · Science & research
SciBench
Can a model solve open-ended, college-level science problems that require multi-step quantitative reasoning, not just recall or short factual answers?
UCLA, Caltech & University of Washington (Wang, Hu, Lu et al.)Released 20 July 2023Live
SciBench pushed back against a limitation the authors saw in earlier science benchmarks: most tested high-school-level content answerable with basic algebra and multiple-choice options, which let a model succeed through pattern-matching rather than working a problem through. Researchers from UCLA, Caltech and the University of Washington instead drew 869 free-response problems from ten college textbooks in physics, chemistry and mathematics — plus smaller multimodal and real-exam sets — each requiring several steps of quantitative reasoning to reach a numerical or symbolic answer, graded on whether that final answer was correct.
The results at release showed how much headroom the harder format opened up. The best configuration tested, GPT-4 combined with a Python interpreter for calculation and given a handful of worked examples, solved only 43.22% of problems, and the authors found that no single prompting technique — chain-of-thought, tool use, self-refinement — reliably helped across all problem types; some improved one subject’s results while hurting another’s. The paper also introduced an error-categorisation protocol asking models to classify their own mistakes across ten reasoning skills, aimed at diagnosing where multi-step scientific reasoning broke down rather than just reporting a single score.
SciBench arrived early in a wave of benchmarks — later joined by SciCode, ChemBench and GPQA — built specifically to resist the fast saturation that had already overtaken multiple-choice science tests, by requiring models to show full quantitative work rather than select an answer from a short list.
The set
869 free-response problems drawn from ten college textbooks in physics, chemistry and mathematics, plus a 177-problem multimodal set and 103 problems from real exams. Answers are numerical or symbolic and require working through a full solution, unlike the multiple-choice format of most prior science benchmarks.
Example
A physics problem from the paper's appendix: "Two charged particles are fixed to an x axis: Particle 1 of charge q1=2.1×10⁻⁸ C is at position x=20 cm and particle 2 of charge q2=−4.00q1 is at position x=70 cm. At what coordinate on the axis (other than at infinity) is the net electric field produced by the two particles equal to zero?" (Answer: −30 cm)arxiv.org
Where it stands
At release the best-performing setup — GPT-4 using few-shot prompting combined with a Python interpreter for calculation — scored 43.22%, and the authors found no single prompting strategy improved every problem type, arguing that scaling model size alone would not close the gap on this kind of multi-step scientific reasoning.