Konwinski Prize

The Konwinski Prize is less a standing benchmark than a pointed experiment about what other coding benchmarks might be measuring. Andy Konwinski, a co-founder of Databricks and Perplexity, pledged $1 million to the first open-source, open-weight AI system to score above 90% on a SWE-bench-style test built from real GitHub issues — with one deliberate twist: the issues used to score entrants were collected only after the submission deadline had passed, so nothing in the test set could have leaked into any entrant’s training data.

That design choice was the point. SWE-bench draws its problems from public repositories that are also, inevitably, used to train the very models being evaluated on them, which leaves open the possibility that a high score reflects memorised answers rather than problem-solving. Konwinski’s prize, run on Kaggle with a guaranteed $50,000 baseline reward regardless of outcome, was built to close off that possibility entirely and see what was left.

What was left was a considerably harder problem than the leaderboards suggested. The first competition closed in mid-2025 with the winning entry solving just 7.5% of tasks, against the roughly 75% models were then routinely posting on SWE-bench Verified and 34% on the harder full SWE-bench set. Konwinski called it “a reality check,” and was careful not to claim the gap definitively proved contamination in the older benchmark rather than simply reflecting how much harder unseen, real-world engineering work is — but either reading left the same practical conclusion: headline SWE-bench scores had been telling a more optimistic story than genuinely novel problems supported.