SWE-bench Pro

SWE-bench Pro is Scale AI’s answer to a problem the original SWE-bench had run into by 2025: frontier models were clearing more than 70% of the “Verified” subset, and it was no longer clear how much of that reflected genuine software-engineering skill versus the benchmark’s own tasks and fixes having leaked into training data. The successor scaled up and toughened the task pool — 1,865 problems across 41 repositories, averaging over a hundred lines of change across multiple files rather than the narrow single-function patches common in the original — and, crucially, kept a large slice of the tasks private, drawn from commercial codebases models were unlikely to have ever seen.

The effect was immediate: models that scored above 70% on SWE-bench Verified managed only around 23% on SWE-bench Pro at launch, a gap Scale read as evidence of overfitting to a familiar, narrow task distribution. Scores climbed through the following year as labs targeted the new benchmark directly — OpenAI’s Codex-tuned models moved from the mid-50s toward 57%, and Anthropic reported Claude Opus 4.7 reaching 64.3% — and OpenAI formally recommended SWE-bench Pro as a successor when it retired Verified from its own release reporting in February 2026.

As with several benchmarks in this space, a gap has opened between what labs report from their own runs and what Scale’s own public leaderboard shows when it reruns submissions independently — the leaderboard has generally sat several points below vendor self-reported figures, a reminder that “SWE-bench Pro score” can mean somewhat different things depending on who ran the test.