Benchmarks / PaperBench
PaperBench
Built by OpenAI · Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, et al. · released 2 Apr 2025
An agent gets one of 20 ICML 2024 Spotlight or Oral papers and must replicate it from scratch: understand the contributions, write a codebase, and run the experiments. Rubrics written with the paper authors break each paper into 8,316 gradable items, and an LLM judge scores the result.
Frontier
26%
Average replication score
IterativeAgent o1-high (36h limit) · harness: IterativeAgent
2 Apr 2025 · Source: OpenAI (benchmark maintainers)
26.0 ± 0.3 over 3 runs with a 36-hour limit. Top of the official leaderboard, which has not been updated since launch. OpenAI reporting its own model.
The best result is at 20–60% of the ceiling, or at 50–100% of human parity.
What it measures
How much of a research paper's empirical results an agent can reproduce with no access to the authors' code. Rubric leaves cover code development, execution, and result match. The headline is the average replication score across the 20 papers. A lighter Code-Dev variant skips execution and grades only code development; its scores are much higher and are not comparable.
Average replication score: Mean rubric score across the 20 papers on the full PaperBench pipeline, judged by SimpleJudge with o3-mini, averaged over 3 runs. Higher is better.
Status compares the frontier with a ceiling of 100%.
Facts
- Grain
- Project level: whole projects judged by an acceptance standard
- Environment
- cli, repo
- Grading
- llm-judge, rubric
- Tasks
- 20
- Human reference
- ML PhDs (best of 3 attempts) scored 41.4% on a 3-paper subset after 48 hours, against 26.6% for o1 on the same subset.
- Contamination
- Agents are told not to use the authors' public code, and a monitor checks for blacklisted repositories. The papers and their code are public, so training data may include them.
- Reuse
- Code and rubrics are in the MIT-licensed openai/frontier-evals repo. Papers belong to their authors. (open-mit)
Limits to keep in mind
- The official leaderboard has not changed since 2025-04-02. Newer models are reported only in lab posts, usually on the Code-Dev variant with different judges, so they are not comparable. Source
- Grading uses an LLM judge. The paper's JudgeEval shows the judge agrees with humans imperfectly, and full grading is expensive. Source
- The human baseline covers only 3 of 20 papers and 8 participants, and models were run under slightly different conditions. Source
- A Qwen3.8 model card lists PaperBench scores of 64.8 to 93.0 for several models, but those are Code-Dev runs judged by Claude Opus 4.6. Aggregators show them as full PaperBench scores. Source
Recorded results
A selection that shows the frontier over time. The full leaderboard is at the source.
| System | Average replication score | Date | Source |
|---|---|---|---|
| BasicAgent claude-3.5-sonnet harness: BasicAgent | 21% | 2 Apr 2025 | OpenAI · primary |
| IterativeAgent o1-high (24h limit) harness: IterativeAgent | 24.4% | 2 Apr 2025 | OpenAI · primary |
| IterativeAgent o1-high (36h limit) · frontier harness: IterativeAgent | 26% | 2 Apr 2025 | OpenAI · primary |
Timeline
- 2 Apr 2025 — OpenAI releases PaperBench for replicating ICML papers. Source
Where it sits in the atlas
Go to the source
- Website openai.com
- Paper arxiv.org
- Full leaderboard github.com
- Code github.com
Last checked 23 Sep 2026 against 5 primary sources. See an error? Tell us.
How to cite
Credit the original work first: PaperBench by OpenAI (https://arxiv.org/abs/2504.01848).
Then, if you used this page:
Can Agents Work. "PaperBench: frontier results and sources." https://canagentswork.com/benchmarks/paperbench/ (accessed 2026-09-24). CC BY 4.0.@misc{caw-paperbench,
title = {{PaperBench: frontier results and sources}},
author = {{Can Agents Work}},
year = {2026},
howpublished = {\url{https://canagentswork.com/benchmarks/paperbench/}},
note = {Accessed 2026-09-24. CC BY 4.0}
}