Can Agents Work?

Benchmarks / AIDev

AIDev

Built by Queen's University · Hao Li, Haoxiang Zhang, Ahmed E. Hassan · released 20 Jul 2025

A public dataset of pull requests that autonomous coding agents (OpenAI Codex, Devin, GitHub Copilot, Cursor, Claude Code) opened on GitHub, with review timelines, code changes, and merge outcomes. The paper compares agent PRs with human PRs in popular repositories.

Frontier

65.3%

PR acceptance rate (AIDev-pop)

OpenAI Codex

20 Jul 2025 · Source: Queen's University (benchmark maintainers)

Table 5, AIDev-pop. 21.3% rejected. Accepted Codex PRs closed in a median 0.3 hours versus 3.9 hours for human PRs. Human baseline 76.8% accepted. Body text gives 64% from Figure 3.

Unrated

No fixed reference point (for example Elo scores or field signals).

AIDev: PR acceptance rate (AIDev-pop) over time, 5 recorded results. 0%20%40%60%80%Jul 2025Jul 2025Jul 2025Jul 2025Aug 2025 OpenAI Codex: 65.3% (20 Jul 2025) Claude Code: 52.5% (20 Jul 2025) Cursor: 51.4% (20 Jul 2025) Devin: 48.9% (20 Jul 2025) GitHub Copilot: 38.2% (20 Jul 2025)
Dots are recorded results; the line is the best result so far; the red dot is the current frontier. Sources for every point are in the table below.
Embed this card

The image names the institutions behind the numbers and links back to this page.

HTML:

<a href="https://canagentswork.com/benchmarks/aidev-dataset/"><img src="https://canagentswork.com/og/benchmarks-aidev-dataset.png" width="600" height="315" alt="AIDev: the best result is 65.3% (OpenAI Codex, 20 Jul 2025)." loading="lazy"></a>

Markdown:

[![AIDev: the best result is 65.3% (OpenAI Codex, 20 Jul 2025).](https://canagentswork.com/og/benchmarks-aidev-dataset.png)](https://canagentswork.com/benchmarks/aidev-dataset/)

What it measures

How agent-authored pull requests fare in real open-source review. The headline is the acceptance rate: the share of PRs in the AIDev-pop subset (popular repositories) that were merged, per agent, against a human baseline from the same repositories. The paper also reports task type mix and review turnaround. The paper covers 456,535 agent PRs in 61,453 repositories (cutoff 2025-06-22). Later releases are larger (v4: 2,743,854 PRs from six agents to November 2025) but publish no acceptance table comparable to Table 5.

PR acceptance rate (AIDev-pop): Share of an agent's pull requests in popular repositories that were merged, from Table 5 of the paper. Higher is better.

No fixed reference point, so we do not rate its status. Field signal from observational data. The human baseline of 76.8% is shown for context, not as a parity line.

Facts

Grain
Task level: bounded tasks with a clear spec
Environment
field-data
Grading
field-signal
Human reference
Human-authored PRs in the same AIDev-pop repositories: 76.8% accepted, 16.1% rejected (Table 5).
Reuse
CC BY 4.0 on the Hugging Face dataset card. The card says that content from the source repositories keeps its original licenses (MIT, Apache-2.0, GPL family, and others). (cite-only)

Limits to keep in mind

  • Observational data. Agents are used by different people on different repositories, so acceptance rates mix agent quality with user selection and repository norms. Source
  • Agents are found by GitHub search patterns (for example head:codex/, Co-Authored-By: Claude), which miss PRs that do not carry the pattern. Cursor and Claude Code are mostly used locally, so their GitHub footprint is small. Source
  • The paper's numbers were captured about two months after several agents launched (Codex started 2025-05-16). The authors say the gaps show how early the transition is. Source
  • The paper defines AIDev-pop as repositories with more than 500 stars; the repo README says more than 100 stars and gives different PR counts. We use the paper's Table 5. Source

Recorded results

A selection that shows the frontier over time. The full leaderboard is at the source.

SystemPR acceptance rate (AIDev-pop)DateSource
OpenAI Codex · frontier65.3%20 Jul 2025Queen's University · primary
Claude Code52.5%20 Jul 2025Queen's University · primary
Cursor51.4%20 Jul 2025Queen's University · primary
Devin48.9%20 Jul 2025Queen's University · primary
GitHub Copilot38.2%20 Jul 2025Queen's University · primary

Timeline

  • 21 Aug 2026 — AIDev v4 grows the dataset to 2,743,854 agent pull requests from six agents. Source
  • 20 Jul 2025 — AIDev dataset of agent pull requests released. Source

Where it sits in the atlas

Work ladder: Direct evidence for Software engineering, but its metric is not a share of tasks completed. It adds to the work measured, but it cannot set a step. How the ladder works

Work it measures (O*NET work activities): Design computer or information systems or applications; Program computer systems or production equipment.

Software engineering

Shipping pull requests

Last checked 10 Oct 2026 against 7 primary sources. See an error? Tell us.

How to cite

Credit the original work first: AIDev by Queen's University (https://arxiv.org/abs/2507.15003).

Then, if you used this page:

Can Agents Work. "AIDev: frontier results and sources." https://canagentswork.com/benchmarks/aidev-dataset/ (accessed 2026-10-11). CC BY 4.0.
@misc{caw-aidev-dataset,
  title        = {{AIDev: frontier results and sources}},
  author       = {{Can Agents Work}},
  year         = {2026},
  howpublished = {\url{https://canagentswork.com/benchmarks/aidev-dataset/}},
  note         = {Accessed 2026-10-11. CC BY 4.0}
}