Software engineering on real repositories · Contested (since 2025)

Coding agents built on frontier models resolve more than 70 percent of SWE-bench Verified issues (Claude Opus 4 at 72.5 percent in May, GPT-5 at 74.9 percent in August), up from low single digits when the benchmark appeared in 2023.

SWE-bench tasks are real GitHub issues with hidden tests, which is closer to the job than any earlier benchmark and still far from it: the issue is already written, the repository is popular and present in training data, and the tests decide. In February 2026 OpenAI stopped reporting SWE-bench Verified, citing contamination and flawed tests, and recommended the harder SWE-bench Pro, where scores were much lower. Agents now write a large share of production code under human review. Owning a codebase over years, deciding what to build and being on call for it are contested at the edges and not yet given up.