Can Agents Work?

Benchmarks / SWE-bench Verified

SWE-bench Verified

Built by Princeton University and OpenAI · released 13 Aug 2024

A human-validated subset of 500 SWE-bench tasks. Each task is a real GitHub issue from one of 12 Python repositories. The agent gets the repository and the issue text and must produce a patch. OpenAI and the SWE-bench team built it in 2024 after 93 developers screened 1,699 SWE-bench tasks for unfair tests and vague issue descriptions.

Frontier

79.2%

Resolved

Sonar Foundation Agent + Claude 4.5 Opus · harness: Sonar Foundation Agent · agent: Sonar Foundation Agent

5 Dec 2025 · Source: Princeton University (benchmark maintainers)

Tied at 79.2% with live-SWE-agent + Claude 4.5 Opus (medium), dated 2025-12-15. Neither entry is checked by the SWE-bench team. Single attempt.

Retired

Maintainers or a major user stopped using it as a frontier measure.

Retired on 23 Feb 2026: OpenAI stopped reporting SWE-bench Verified on 2026-02-23. Its audit found that at least 59.4% of 138 hard tasks had tests that reject correct solutions, and that all frontier models it probed could reproduce gold patches or problem text from training data. OpenAI told other developers to stop too. The swebench.com leaderboard still exists but has few entries after early 2026. Source.

See the full leaderboard at Princeton University

SWE-bench Verified: Resolved over time, 5 recorded results. 0%20%40%60%80%100%Oct 2024Jan 2025Apr 2025Jul 2025Oct 2025Jan 2026 GPT-4o (best open-source scaffold): 33.2% (13 Aug 2024) OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022): 53% (29 Oct 2024) TRAE + Claude Sonnet 4 + Opus 4 + Sonnet 3.7 + Gemini 2.5 Pro: 75.2% (12 Jun 2025) Sonar Foundation Agent + Claude 4.5 Opus: 79.2% (5 Dec 2025) Claude 4.5 Opus (high), mini-SWE-agent: 76.8% (17 Feb 2026)
Dots are recorded results; the line is the best result so far; the red dot is the current frontier. Sources for every point are in the table below.
Embed this card

The image names the institutions behind the numbers and links back to this page.

HTML:

<a href="https://canagentswork.com/benchmarks/swe-bench-verified/"><img src="https://canagentswork.com/og/benchmarks-swe-bench-verified.png" width="600" height="315" alt="SWE-bench Verified: the best result is 79.2% (Sonar Foundation Agent + Claude 4.5 Opus, 5 Dec 2025)." loading="lazy"></a>

Markdown:

[![SWE-bench Verified: the best result is 79.2% (Sonar Foundation Agent + Claude 4.5 Opus, 5 Dec 2025).](https://canagentswork.com/og/benchmarks-swe-bench-verified.png)](https://canagentswork.com/benchmarks/swe-bench-verified/)

What it measures

Share of the 500 issues resolved: the patch must make the hidden fail-to-pass tests pass and keep the pass-to-pass tests passing. The official leaderboard accepts submissions from any agent system, so scores mix model quality with scaffold quality. A "Bash Only" view runs every model in the same mini-SWE-agent environment.

Resolved: Percentage of the 500 task instances whose tests pass after the agent's patch is applied. Higher is better.

Status compares the frontier with a ceiling of 100%.

Facts

Grain
Task level: bounded tasks with a clear spec
Environment
repo, cli
Grading
automated-tests
Tasks
500
Human reference
Annotators estimated task difficulty by human time. 196 tasks are under 15 minutes, 45 tasks are over 1 hour. No direct human success rate is published.
Contamination
High. Tasks and gold patches are public since 2024 and the source repositories are common in training data. OpenAI's red-team probes got GPT-5.2-Chat, Claude Opus 4.5, and Gemini 3 Flash Preview to reproduce gold patches or verbatim task details.
Reuse
Task content comes from 12 open-source Python repositories under their own licenses. The SWE-bench harness is MIT. (cite-only)

Limits to keep in mind

  • OpenAI's audit of 138 tasks that o3 often failed found 59.4% had material problems: 35.5% had tests tied to one implementation, 18.8% tested behavior not in the issue. Source
  • Leaderboard entries use different agents, retry budgets, and model mixes. Most top entries are not checked by the SWE-bench team. Source
  • Python only, from 12 repositories, so it does not cover most languages or codebases. Source

Recorded results

A selection that shows the frontier over time. The full leaderboard is at the source.

SystemResolvedDateSource
Claude 4.5 Opus (high), mini-SWE-agent
harness: mini-SWE-agent
76.8%17 Feb 2026Princeton University · primary
Sonar Foundation Agent + Claude 4.5 Opus · frontier
harness: Sonar Foundation Agent
79.2%5 Dec 2025Princeton University · primary
TRAE + Claude Sonnet 4 + Opus 4 + Sonnet 3.7 + Gemini 2.5 Pro
harness: TRAE
75.2%12 Jun 2025Princeton University · primary
OpenHands + CodeAct v2.1 (claude-3-5-sonnet-20241022)
harness: OpenHands
53%29 Oct 2024Princeton University · primary
GPT-4o (best open-source scaffold)33.2%13 Aug 2024OpenAI · primary

Timeline

  • 23 Feb 2026 — OpenAI stops reporting SWE-bench Verified. Source
  • 13 Aug 2024 — OpenAI and the SWE-bench team release SWE-bench Verified. Source

Where it sits in the atlas

Software engineering

Issue resolution

Last checked 23 Sep 2026 against 5 primary sources. See an error? Tell us.

How to cite

Credit the original work first: SWE-bench Verified by Princeton University and OpenAI (https://arxiv.org/abs/2310.06770).

Then, if you used this page:

Can Agents Work. "SWE-bench Verified: frontier results and sources." https://canagentswork.com/benchmarks/swe-bench-verified/ (accessed 2026-09-24). CC BY 4.0.
@misc{caw-swe-bench-verified,
  title        = {{SWE-bench Verified: frontier results and sources}},
  author       = {{Can Agents Work}},
  year         = {2026},
  howpublished = {\url{https://canagentswork.com/benchmarks/swe-bench-verified/}},
  note         = {Accessed 2026-09-24. CC BY 4.0}
}