DeepSWE

By 2026, the two problems dogging coding benchmarks built from real GitHub history were well known: models may have already seen the fix during pretraining, since merged pull requests sit in public data, and grading against one specific reference patch penalises a model for solving the problem a different but equally valid way. DeepSWE, built by the data-labelling firm Datacurve, was designed to close both gaps at once. Its 113 tasks span 91 open-source repositories in five languages, but none of the reference solutions were ever merged upstream — they were written from scratch for the benchmark, so a model cannot have trained on the fix — and each is graded by a hand-written verifier that checks behaviour rather than requiring a specific implementation.

That grading approach produced a striking gap in reliability: the paper reports human graders disagreeing with the automated verifier only 1.4% of the time, against 32.4% disagreement on SWE-bench Pro, a benchmark that grades primarily by matching hidden test suites. DeepSWE’s reference solutions also touch roughly 5.5 times more code than typical benchmark tasks despite shorter prompts, reflecting an emphasis on substantial, multi-file engineering work rather than narrowly scoped bug fixes.

DeepSWE arrived alongside a wave of similarly-named and similarly-motivated efforts addressing the same saturation and contamination problems that had begun to affect SWE-bench and its “Verified” subset by 2026. Its name is shared, confusingly, with an unrelated 2025 open-source coding agent from Agentica and Together AI — a model, not a benchmark — a collision worth noting for anyone cross-referencing results under the same name.