SWE-bench

SWE-bench asks the question that a coding demo cannot: not whether a model can write plausible code, but whether it can fix a real problem in a real, large codebase that it has never seen. Each task is an actual issue filed against a popular open-source Python project, paired with the pull request that eventually resolved it. The model is handed the whole repository and the issue text, and must produce a patch — the patch counts only if the project’s own test suite, hidden from the model, passes afterwards.

When Princeton and Stanford researchers released it in 2023, the best available system solved under 2% of tasks, and the benchmark looked almost hopeless. What followed became one of the clearest illustrations of how fast agentic coding advanced: Cognition’s Devin cleared 13.86% in early 2024, and by late 2025 frontier models were resolving more than three-quarters of the human-screened “Verified” subset that OpenAI introduced to clean up the original set’s ambiguous tasks.

That progress is also the benchmark’s problem. As scores approached saturation, attention shifted to harder measures — Scale AI’s SWE-bench Pro, built from unseen commercial code, dropped the same top models to around 23% — and researchers cautioned that a passing patch is not the same as a mergeable one: METR found many SWE-bench-passing pull requests would be rejected by human reviewers. Even so, SWE-bench remains the single most-cited yardstick for whether an AI system can do software engineering rather than merely talk about it.