Scale AI launches SWE-bench Pro
The leading models scored around 23%, against over 70% on the older SWE-bench Verified, a gap Scale AI attributed to unseen, real-world commercial codebases.
- Benchmarks & progress
- Notable
Scale AI released SWE-bench Pro, a coding benchmark built to replace SWE-bench Verified once frontier models had driven that benchmark close to saturation. The new set comprised 1,865 tasks — 731 public, 858 held out, and 276 drawn from private commercial codebases — spanning 41 repositories, with problems averaging 107 lines of code across several files rather than the narrow single-function fixes common in earlier coding benchmarks.
The design targeted two weaknesses Scale said had let models game SWE-bench Verified: memorised training data and artificially simple problems. Tasks were sourced from GPL-licensed and private repositories models were unlikely to have seen during training, drawn from consumer applications, business software and developer tools rather than isolated utility libraries, and left deliberately ambiguous in the way real bug reports and feature requests are.
The effect on scores was immediate. OpenAI’s GPT-5 and Anthropic’s Claude Opus 4.1 — both scoring above 70% on SWE-bench Verified — managed only around 23% on SWE-bench Pro, a gap Scale presented as evidence that Verified’s saturation reflected overfitting to a known, narrow task distribution rather than genuine software-engineering competence.
The benchmark’s contamination-resistant design and harder task pool made it an early candidate to succeed SWE-bench Verified as the field’s reference coding evaluation, which OpenAI confirmed the following February when it recommended SWE-bench Pro among Verified’s successors on retiring the older benchmark from its own reporting.