SWE-bench Pro Leaderboard - Scale AI

SWE-Bench Pro represents a significant advancement in AI agent evaluation for software engineering tasks. By addressing critical limitations in existing benchmarks, such as data contamination, limited task diversity, and oversimplified problems, the benchmark offers a more authentic assessment of AI problem-solving capabilities. The methodology involves a sophisticated four-stage workflow that carefully sources, creates, and augments software engineering challenges from diverse repositories. The benchmark's key innovation lies in its rigorous design, which includes three distinct dataset subsets: a public set, a commercial set, and a held-out set. This approach allows for comprehensive testing across different coding environments and provides a more nuanced understanding of AI agents' generalization abilities. The results are striking, with top models like OpenAI GPT-5 and Claude Opus 4.1 scoring only around 23% on the public dataset, compared to 70%+ on previous benchmarks. This dramatic performance drop highlights the benchmark's increased complexity and its potential to drive meaningful improvements in AI software engineering capabilities.