Sources and licenses
Every benchmark, who made it, how you may reuse its data, and when we last checked it. Our own contributions are CC BY 4.0; source data keeps its original terms. Institutions · Downloads.
Benchmarks
| Benchmark | Built by | Reuse | Last checked |
|---|---|---|---|
| Aider Polyglot leaderboard | Aider | Cite; link to the source Exercises are copyright Exercism and used under the Exercism tracks' open-source licenses. The benchmark repository states no license of its own. | 10 Oct 2026 |
| AIDev | Queen's University | Cite; link to the source CC BY 4.0 on the Hugging Face dataset card. The card says that content from the source repositories keeps its original licenses (MIT, Apache-2.0, GPL family, and others). | 10 Oct 2026 |
| AIOpsLab | Microsoft Research, UIUC, UC Berkeley, and IISc | Open (MIT) MIT (GitHub repository license). | 10 Oct 2026 |
| AISI multi-step cyber attack ranges | AI Security Institute | Cite; link to the source The ranges are private and were built for AISI by SpecterOps and Hack The Box. The paper describes the attack chains and asks to be excluded from training data (canary GUID). No data release. | 10 Oct 2026 |
| Ambig-SWE | Carnegie Mellon University | Open (MIT) Paper CC BY 4.0; repo MIT. Issues derive from SWE-Bench Verified. | 10 Oct 2026 |
| APEX-Accounting | Mercor and Ramp | Cite; link to the source The 160 scored tasks are private and never released. A public dev set (1 world, 10 tasks) is on Hugging Face under CC BY 4.0. The tool layer (accounting software interface) is not released. | 10 Oct 2026 |
| APEX-Agents | Mercor, Box, and Harvey | Open (CC BY) CC BY 4.0 on the Hugging Face dataset. Mercor says the full task set used for the leaderboard stays private. | 10 Oct 2026 |
| ARTEMIS live penetration test (agents vs. professionals) | Stanford University, Carnegie Mellon University, and Gray Swan AI | Cite; link to the source Study artifacts are described as released with the code, but we found no data license. The target is a live network and cannot be re-run. The ARTEMIS repository is Apache-2.0. | 10 Oct 2026 |
| AutomationBench | Zapier | Open (MIT) The 600 public tasks (100 per domain) and the harness ship in the repository under the MIT license. The private leaderboard set is not released. | 10 Oct 2026 |
| BEHAVIOR Challenge 2025 | Stanford University | Open (MIT) MIT (Hugging Face dataset card for behavior-1k/2025-challenge-demos). | 10 Oct 2026 |
| BIRD-Interact | HKU and Google Cloud | Open (other license) CC BY-SA 4.0 (README). Task data is on Hugging Face; the gold SQL and test cases are sent by email on request to limit crawling. | 10 Oct 2026 |
| BrowseComp | OpenAI | Open (MIT) Released in OpenAI's simple-evals repository, which lists BrowseComp under the MIT License. | 10 Oct 2026 |
| Code Review Bench | Martian | Open (MIT) MIT (repo license covers PRs metadata, golden comments, judge prompts, pipeline, and results) | 10 Oct 2026 |
| CodeClash | Stanford University and Princeton University | Open (MIT) MIT (CodeClash repository). Arenas are third-party games with their own licenses. | 10 Oct 2026 |
| CORE-Bench | Princeton University | Open (MIT) Harness and code are MIT licensed. Papers and capsules come from CodeOcean and keep their own licenses. | 10 Oct 2026 |
| CRMArena-Pro | Salesforce AI Research | Cite; link to the source CC BY-NC 4.0 (repository LICENSE.txt). Research use only. | 10 Oct 2026 |
| CVE-Bench | UIUC | Open (Apache-2.0) Apache-2.0 (GitHub repository license). Reference exploits are available on request. | 10 Oct 2026 |
| Cybench | Stanford University | Open (Apache-2.0) Tasks are public in the GitHub repository (Apache-2.0); the site asks users to cite the ICLR 2025 paper. | 10 Oct 2026 |
| CyberBench v1.1 | Vals AI | Cite; link to the source Private task set ("dataset_type: private" in the page data). No license stated. | 10 Oct 2026 |
| CyberGym | UC Berkeley | Cite; link to the source Not stated. The Hugging Face dataset card (about 240 GB) gives no license, and the tasks come from OSS-Fuzz projects under their own licenses. | 10 Oct 2026 |
| DARPA AI Cyber Challenge, Final Competition | DARPA and ARPA-H | Cite; link to the source DARPA says all seven CRSs are released under an OSI-approved open-source license and that the competition challenges, framework, and telemetry will also be open sourced (archive.aicyberchallenge.com). We did not read the licenses of the individual releases. | 10 Oct 2026 |
| EnterpriseOps-Gym | ServiceNow, Mila, and Université de Montréal | Open (Apache-2.0) Apache 2.0 on the Hugging Face dataset card and the repository. The public split holds 60% of the tasks (649 rows in the oracle config); the rest is private. | 10 Oct 2026 |
| Finance Agent Benchmark | Vals AI | Cite; link to the source 50 public validation questions in the repository (MIT). 150 validation questions are available for license. The 337-question test set is private. | 10 Oct 2026 |
| GAIA | Meta and Hugging Face | Cite; link to the source Gated dataset on Hugging Face. Test answers are private. The maintainers ask that the public set not be reposted or used for training. | 10 Oct 2026 |
| GDPval | OpenAI | Cite; link to the source The 220-task gold subset (prompts and reference files) is public on Hugging Face. The other 1,100 tasks and the expert grades are private. Rubrics and gold deliverables were later released. | 10 Oct 2026 |
| GDPval-AA | Artificial Analysis | Cite; link to the source Tasks are OpenAI's public GDPval gold set. The Stirrup harness is open source on GitHub. Artificial Analysis publishes Elo scores and example submissions on its site. | 10 Oct 2026 |
| Harvey Legal Agent Benchmark | Harvey and Vals AI | Open (MIT) Public tasks, rubrics, and harness are in the MIT-licensed repository. The held-out set used for scores is private. | 10 Oct 2026 |
| HealthAgentBench | Microsoft Research | Cite; link to the source Code and task definitions are MIT (repository LICENSE). Gold labels are not in the repository; the run fetches them. Three categories need gated source data with per-user credentials: MIMIC-CXR (PhysioNet), CT-RATE (Hugging Face, OpenRAIL gated), and EHRSHOT (Stanford Redivis). The other categories use public data (CAMELYON16, MIMIC-IV demo, TREC-CT 2021). | 10 Oct 2026 |
| HWE-Bench | Peking University | Open (Apache-2.0) The task dataset on Hugging Face (henryen/hwe-bench) is tagged Apache 2.0. The code is Apache 2.0. Docker images for five projects are public; OpenTitan images are not distributed because that flow needs a Synopsys VCS license. | 10 Oct 2026 |
| IaC-Eval | University of Michigan and Cisco Research | Open (CC BY) CC-BY-4.0 (Hugging Face dataset card) | 10 Oct 2026 |
| ImpossibleBench | Carnegie Mellon University and Anthropic | Open (MIT) Paper CC BY 4.0; repo MIT. Task data derives from SWE-bench Verified and LiveCodeBench. | 10 Oct 2026 |
| ITBench | IBM Research and UIUC | Open (Apache-2.0) Apache-2.0 (repository LICENSE file). Scenario tooling and sample scenarios are open source; hosted environments require registration. | 10 Oct 2026 |
| ITBench-AA | Artificial Analysis and IBM Research | Cite; link to the source 40 public tasks are on Hugging Face under CC BY 4.0. The 19 held-out tasks, shared by the ITBench team, are private to Artificial Analysis. | 10 Oct 2026 |
| Legal Research Bench | Vals AI | Cite; link to the source The page labels the benchmark "Proprietary". 5 public sample questions and the agent harness are open in the repository (MIT). 200 validation questions are available for license. The 208-question test set is private. | 10 Oct 2026 |
| MCP Atlas | Scale AI | Open (CC BY) The 500 public tasks are on Hugging Face under CC BY 4.0 (prompts, enabled tools, claims, and reference trajectories). The other 500 tasks are private. | 10 Oct 2026 |
| MCPMark Verified | EVAL SYS, LobeHub, and NUS TRAIL | Open (Apache-2.0) Tasks, initial states, and verifiers ship in the GitHub repository under Apache License 2.0. | 10 Oct 2026 |
| MedAgentBench | Stanford University | Cite; link to the source Code is MIT (repository license). The arXiv paper is CC BY 4.0. Patient data derives from de-identified Stanford STARR records and ships as a Docker image; the reference solutions are distributed separately from a Stanford Medicine Box link. | 10 Oct 2026 |
| METR Task-Completion Time Horizons | METR | Cite; link to the source Horizon estimates and run data are public in eval-analysis-public (no license file seen). Many tasks are private. | 10 Oct 2026 |
| MLE-bench | OpenAI | Cite; link to the source Code is MIT licensed. Competition data comes from Kaggle and users must accept each competition's rules to download it. | 10 Oct 2026 |
| OSWorld 2.0 | HKU and Snorkel AI | Cite; link to the source The task classes are a gated Hugging Face dataset (xlangai/osworld_v2_tasks) whose card says Apache-2.0; the task assets (xlangai/osworld_v2_assets_gated) are gated with no license on the card. The code repository is Apache-2.0. | 10 Oct 2026 |
| OSWorld-Verified | HKU, Salesforce AI Research, Carnegie Mellon University, and University of Waterloo | Open (Apache-2.0) Apache 2.0 on the GitHub repository. Windows tasks need a licensed image. | 10 Oct 2026 |
| PaperBench | OpenAI | Open (MIT) Code and rubrics are in the MIT-licensed openai/frontier-evals repo. Papers belong to their authors. | 10 Oct 2026 |
| Parametric CAD Bench V3 | gNucleus AI | Cite; link to the source The V3 task suite is distributed through Harbor Hub (gnucleus-ai/cad-bench@v3); we found no license text for it. The reference CAD models are held back in a separate verifier environment. The verifier (freecad-validator) is Apache 2.0. The V1 and V2 results archives on Hugging Face are Apache 2.0. | 10 Oct 2026 |
| PhysicianBench | Stanford University | Cite; link to the source Code, tasks, and the evaluation harness are Apache 2.0 (repository LICENSE). The patient records ship as a Docker image on Stanford Redivis under a controlled request-and-approval process, governed by the Stanford University Research Data Use Agreement. | 10 Oct 2026 |
| PR Arena | aavetis | Unknown; cite only No license file in the repo. Data is derived from public GitHub search counts. | 10 Oct 2026 |
| RE-Bench | METR | Open (MIT) Repo is MIT licensed. Reference solutions are password-protected. METR asks users to keep the tasks out of training data and not to publish solutions. | 10 Oct 2026 |
| RedlineBench | Crosby Legal and micro1 | Open (CC BY) CC-BY-4.0 (Hugging Face dataset card and repository README). | 10 Oct 2026 |
| Remote Labor Index | CAIS and Scale AI | Cite; link to the source Scores come from 230 private projects. 10 public projects and the open-source evaluation platform are released for qualitative analysis. | 10 Oct 2026 |
| RoboArena | UC Berkeley and Stanford University | Open (MIT) MIT (Hugging Face card for RoboArena/DataDump_07-17-2026). Paper CC BY 4.0 on arXiv. | 10 Oct 2026 |
| RoboChallenge Table30 | Dexmal and Hugging Face | Cite; link to the source No license on the Table30 dataset card (Hugging Face reports empty metadata). The inference client repo is CC BY-NC-SA 4.0. The report is under the arXiv non-exclusive license. | 10 Oct 2026 |
| SOP-Bench | Amazon | Open (other license) CC BY-NC 4.0 on the repository LICENSE and the Hugging Face dataset card. No commercial use. | 10 Oct 2026 |
| Spider 2.0 | HKU and Salesforce AI Research | Open (MIT) Repository is MIT licensed. All examples and gold answers were released for self-evaluation in December 2024. Maintainers ask users not to fine-tune on the gold SQL. | 10 Oct 2026 |
| SpreadsheetBench | Renmin University of China | Open (other license) CC BY-SA 4.0, stated in the paper's maintenance plan and the repo README. | 10 Oct 2026 |
| SpreadsheetBench 2 | Renmin University of China, Aptura, AfterQuery, and Shortcut | Open (MIT) MIT on the Hugging Face dataset card (KAKA22/SpreadsheetBench-v2). The code repository has no license file. | 10 Oct 2026 |
| SREGym | UIUC and University of Toronto | Open (MIT) MIT (GitHub repository license). Paper is CC BY 4.0. | 10 Oct 2026 |
| SWE-Bench Pro | Scale AI | Cite; link to the source Public-set tasks come from GPL-licensed repositories and stay under those licenses. The harness is MIT. The held-out and commercial sets are not released. | 10 Oct 2026 |
| SWE-Bench Pro V2 | Scale AI and Reflection | Cite; link to the source Tasks come from GPL-licensed repositories and stay under those licenses. The Hugging Face dataset card states no license. The harness repository is MIT. | 10 Oct 2026 |
| SWE-bench Verified | Princeton University and OpenAI | Cite; link to the source Task content comes from 12 open-source Python repositories under their own licenses. The SWE-bench harness is MIT. | 10 Oct 2026 |
| SWE-Lancer | OpenAI | Open (MIT) Diamond split is public in the openai/frontier-evals repository (MIT), formerly openai/preparedness. The remaining tasks are a private holdout. | 10 Oct 2026 |
| SWE-rebench | Nebius | Cite; link to the source Tasks come from repositories under permissive licenses (MIT, Apache-2.0, BSD, ISC, and others that Nebius checked by hand). The leaderboard dataset and Docker images are public. | 10 Oct 2026 |
| Tax Agent Bench | Vals AI | Cite; link to the source 391 questions in total: 5 public samples in the MIT-licensed repository, 193 validation questions available for license, and a 193-question test set that Vals says will stay private. | 10 Oct 2026 |
| Terminal-Bench 2.0 | Laude Institute and Stanford University | Open (Apache-2.0) Apache-2.0 (terminal-bench-2 repository). | 10 Oct 2026 |
| Terminal-Bench 4.0 | Laude Institute and Stanford University | Open (Apache-2.0) Apache-2.0 LICENSE at the root of the terminal-bench repository, which holds the tasks. No separate data license is stated. | 10 Oct 2026 |
| TheAgentCompany | Carnegie Mellon University | Open (MIT) MIT license on the GitHub repository. Task images, evaluators, and company data are public. | 10 Oct 2026 |
| TutorMoments-Preview | Ai2 | Open (CC BY) CC-BY-4.0 on the Hugging Face dataset card (462 de-identified transcripts, annotations, and the 7,280 model replays). | 10 Oct 2026 |
| Vending-Bench 2 | Andon Labs | Cite; link to the source No public code or data release is linked from the benchmark page. Andon Labs runs the simulation. | 10 Oct 2026 |
| Waymo Safety Impact data hub | Waymo | Cite; link to the source No license stated on the hub or in the CSV downloads. | 10 Oct 2026 |
| Will It Survive? Agent code survival study | Concordia University | Cite; link to the source Paper CC BY 4.0. Underlying data comes from the AIDev dataset. | 10 Oct 2026 |
| WorkArena | ServiceNow | Open (Apache-2.0) Apache 2.0 for the benchmark code. Running tasks needs access to a ServiceNow developer instance. | 10 Oct 2026 |
| τ²-bench | Sierra | Open (MIT) MIT (repository license covers tasks, policies, and code). | 10 Oct 2026 |
Where sources disagree
AIOpsLab · resolved · opened 10 Oct 2026
The AIOpsLab paper gives Flash 59.32% accuracy across all problems. The maintainers' leaderboard gives "FLASH (GPT-4)" an average of 59.27. The same leaderboard ranks OpsAgent (GPT-4) first at 78.75. We show the paper value.
- 59.32 — arxiv.org (primary) · Paper Table 3, accuracy across all problems, read again 2026-10-10.
- 59.27 — microsoft.github.io (primary) · Leaderboard row "FLASH (GPT-4)", Average 59.27 with Detection 100, Localization 46.15, Diagnosis 36.36, and Mitigation 54.55. Seen 2026-10-10.
- 78.75 — microsoft.github.io (primary) · Leaderboard row "OpsAgent (GPT-4)", AIOpsLab & NKU, added 2025-06-10, with no linked paper. Its four task scores (100, 60, 83.34, 66.67) average 77.50, not 78.75. Seen 2026-10-10.
We show 59.32. Resolution: Found 2026-10-10: the leaderboard's Average is the plain mean of the four task-type accuracies (for FLASH, (100 + 46.15 + 36.36 + 54.55) / 4 = 59.27), while the paper counts the share of all problems, where root cause analysis has more problems than the other tasks. The leaderboard's four Flash task scores equal the paper's Table 4, so the two numbers aggregate the same results in different ways. The atlas keeps the paper's metric. OpsAgent's 78.75 uses the leaderboard's aggregation, does not match its own task scores, and has no public write-up, so it is not shown as the frontier.
AISI multi-step cyber attack ranges · open · opened 10 Oct 2026
AISI's paper estimates that a human expert would need about 14 hours to complete "The Last Ones". AISI's blog posts from April 2026 on give about 20 hours for the same range. Both are estimates, not timed trials. We did not find a statement from AISI that explains the change, so the cause is open. We show both in human_reference, with the paper's per-step figure (best Opus 4.6 run worth about 6 of 14 hours) tied to the 14-hour estimate.
- about 14 hours — arxiv.org (primary) · arXiv v3 (2026-03-17), Section 2, read 2026-10-10.
- about 20 hours — www.aisi.gov.uk (primary) · AISI blog post dated 2026-04-13, read 2026-10-10.
- around 20 hours — www.aisi.gov.uk (primary) · AISI blog post dated 2026-04-30, read 2026-10-10. The Kimi K3 post (2026-07-23) also says "roughly 20 hours".
We show about 14 hours (paper) and about 20 hours (later AISI posts).
DARPA AI Cyber Challenge, Final Competition · resolved · opened 24 Sep 2026
DARPA's result page says the systems "patched 68% of the vulnerabilities identified". The systems found 54 of 63 synthetic vulnerabilities and patched 43. 43 of 63 is 68%; 43 of the 54 identified would be 79.6%. The sentence and the number do not match. The same page's editor's note ties 68% to the 63 total. We show the counts (54 found, 43 patched, of 63) and not the percent.
- 68% of the vulnerabilities identified were patched (43 of 54 would be 79.6%) — www.darpa.mil (primary) · Body text of the DARPA results page, dated Aug. 8, 2025, read 2026-09-24.
- 68% patched, of the 63 synthetic vulnerabilities (43 of 63) — www.darpa.mil (primary) · Editor's note on the same page: the total was corrected from 70 to 63, "which means competitors discovered 86% of the synthetic vulnerabilities and patched 68%". The same note is on aicyberchallenge.com.
We show 43 patched of 63 (counts only). Resolution: The editor's note shows that 68% is 43 of 63, and that the phrase "of the vulnerabilities identified" in the body is loose wording. We show the raw counts and leave the percent out.
CodeClash · open · opened 23 Sep 2026
The CodeClash site leaderboard shows Claude Sonnet 4.5 at 1385 ± 18 and GPT-5 at 1366 ± 17. The paper shows 1389 ± 18 and 1360 ± 17 for the same models, in v1 (2025-11-02) and again in v2 (2026-05-12). Both come from the maintainers. The gap dates from launch, so it is not a v2 refit. Several per-arena values also differ by a few points (Sonnet 4.5 on Halite: 1413 on the site, 1408 in the paper; on Poker: 1256 and 1248), which points to two separate Elo fits of the tournament results. Neither source explains the difference. The ranking is the same. We show the leaderboard value.
- 1385 — codeclash.ai (primary) · Site leaderboard, "Updated Nov. 3, 2025" (embedded data last_updated 2025-11-03T16:58Z), seen again 2026-10-10.
- 1389 — arxiv.org (primary) · Paper v1 (2025-11-02), Table 1 and the uncertainty table (1389 ± 18).
- 1389 — arxiv.org (primary) · Paper v2 (2026-05-12), Table 1 and Table 4 (1389 ± 18).
We show 1385.
EnterpriseOps-Gym · resolved · opened 24 Sep 2026
Three "best model" numbers appear at the source. The paper says Claude Opus 4.5 reaches 37.4%. The README leaderboard shows 39.4% for the same model, with the same eight domain scores. The README intro says "best model achieves only 34.1%", which equals the board's row for Claude Sonnet 4.5, while the board's top row is Claude Opus 4.6 at 45.9%.
- 37.4 — arxiv.org (primary) · Paper abstract and Table 2 (2026-03-13), Claude Opus 4.5, oracle tool mode.
- 39.4 — github.com (primary) · README leaderboard "Avg" for Claude Opus 4.5, full benchmark, oracle mode.
- 34.1 — github.com (primary) · README intro line "Best model achieves only 34.1% success rate". Matches the board row for Claude Sonnet 4.5.
We show 39.4. Resolution: Different averages of the same domain scores. The README board's Avg is the unweighted mean of the eight domain rates (Opus 4.5: 315.4 / 8 = 39.4; Sonnet 4.5: 34.1). The paper's 37.4 matches a task-weighted mean of the same eight rates using the public split's domain proportions from the dataset card (37.41), where the weaker CSM, ITSM, and HR domains have the most tasks; the paper does not state its weighting, so this part is inferred. The 34.1% intro line is stale. We show the README board, which is the series that contains the current frontier (Claude Opus 4.6, 45.9%).
GDPval-AA · resolved · opened 10 Oct 2026
Anthropic's Claude Opus 5.5 post gives 1846 Elo on GDPval-AA v2.1. The live Artificial Analysis board now shows 1866 for the same entry. Both are v2.1 with the same anchor.
- 1866 — artificialanalysis.ai (primary) · Claude Opus 5.5 (Max, Default Fallback), -26 / +26, board seen 2026-10-10.
- 1846 — www.anthropic.com (lab-reported) · Opus 5.5 at max effort in Anthropic's launch post, which matches the board as we read it on 2026-09-23.
We show 1866. Resolution: Same version, same anchor, different day. The board fits all ratings together with a Crowd-BT model, with DeepSeek V4.1 Flash (max) pinned at 1600, so ratings move when new models and comparisons are added (our reading of the methodology page; Artificial Analysis does not log each refit). Between 2026-09-23 and 2026-10-10 Opus 5.5 moved from 1846 to 1866, and Grok 4.7 (xhigh) now shows 1712 against the 1695 in xAI's post. Lab posts quote the board on their launch day, so we show the live board value and date it by the day we read it.
Harvey Legal Agent Benchmark · open · opened 24 Sep 2026
Harvey's initial results (May 2026) put Claude Opus 4.7 at 7.1% all-pass on the held-out set. Vals AI's board, which uses Harvey's harness and grading protocol on the same set, shows 6.67% for the same model (unchanged on 2026-10-10). The likely cause is separate agent runs and judge sets: Harvey averaged several gradings across model families, while Vals used GPT 5.5 and Claude Sonnet 4.6 as judges after fixing a DOCX tracked-changes bug in the harness. Our own arithmetic, not stated by either source: the Vals page data lists 120 tasks, so with two judges one task judgment is worth 0.417 points; 6.67% is 16 of 240 judgments and 7.1% is about 17 of 240, so the gap is about one judgment. Neither source explains it.
- 7.1 — www.harvey.ai (primary) · Harvey post dated 2026-05-26, Figure 1; text reread 2026-10-10.
- 6.67 — www.vals.ai (third-party) · Page data 6.667 (stderr 1.998 on 2026-09-24), page updated 2026-09-22 and again 2026-10-07, seen 2026-09-24 and 2026-10-10.
We show 7.1.
ITBench · resolved · opened 23 Sep 2026
The ICML 2025 paper says agents resolve 11.4% of SRE scenarios. The arXiv v1 abstract says 13.8%. The official SRE leaderboard shows 25.0% for IBM's GPT-4o reference agent. We show the leaderboard value because it is the maintainers' most recent published result.
- 11.4 — proceedings.mlr.press (primary) · ICML 2025 camera-ready abstract, 102 scenarios. Table 4 gives GPT-4o mitigation pass@1 11.43% on 42 SRE scenarios.
- 13.8 — arxiv.org (primary) · arXiv v1 abstract (2025-02-07), 94 scenarios. Table 4 gives GPT-4o diagnosis pass@1 13.81% on the same 42 SRE scenarios.
- 25 — github.com (primary) · SRE leaderboard, ITBench-SRE-Agent-GPT-4o, 16 trials, updated 2 May 2025. Multi-trial table shows 24.79%. Unchanged on 2026-10-10.
We show 25. Resolution: Found 2026-10-10 in the two paper PDFs. Both versions have the same SRE results table (42 SRE scenarios, 10 runs each). The arXiv v1 abstract quotes GPT-4o's diagnosis rate (13.81%), and the ICML abstract quotes its mitigation rate (11.43%). The scenario totals differ (94 and 102) because the ICML version has 10 FinOps scenarios instead of 2; the SRE set is the same. The leaderboard's 25.0% is a separate run of the reference agent in IBM's hosted environment (16 trials). It also measures repair, on fewer incidents, so it is not directly comparable with the paper; the atlas keeps it as the headline because it is the maintainers' public board.
METR Task-Completion Time Horizons · resolved · opened 23 Sep 2026
METR's Time Horizon 1.1 blog post (2026-01-29) gives Claude Opus 4.5 a 50% horizon of 320 minutes. METR's live data file gives 293.0 minutes. Both are METR sources. The results page lists a 2026-03-03 update that "corrected a regularization mistake that affected our measurements", which explains the change. GPT-5 moved the same way (214 to 203 minutes).
- 293 — metr.org (primary) · Live data file linked from metr.org/time-horizons, retrieved 2026-09-23. CI 161.7 to 623.7.
- 320 — metr.org (primary) · TH1.1 launch post, table "Changes to Model Horizon Estimates". CI 170 to 729.
We show 293. Resolution: We show the live data file, which reflects the 2026-03-03 correction. The blog post predates it.
Remote Labor Index · open · opened 23 Sep 2026
CAIS, a co-author of the benchmark, reports 15.8% for Claude Fable 5, and on 2026-10-10 the Scale leaderboard and the CAIS dashboard both show 15.83. Aggregator pages report 16.1%. The earliest 16.1% we found is a news digest dated 2026-07-05 that quotes a social-media summary of the CAIS post. We did not find a primary source for 16.1%, so the cause is still open.
- 15.8 — safe.ai (primary) · CAIS post dated 2026-07-01, read again 2026-10-10.
- 15.83 — labs.scale.com (primary) · Scale leaderboard row "Fable 5", entry added 2026-07-01, seen 2026-10-10.
- 15.83 — dashboard.safe.ai (primary) · CAIS dashboard, Automation section, row "Fable 5", seen 2026-10-10.
- 16.1 — www.benchleader.com (third-party) · BenchLeader page, seen 2026-09-23 and again 2026-10-10 (all its other RLI rows match the Scale board).
- 16.1 — ai-intensify.com (third-party) · News article, seen 2026-09-23.
- 16.1 — www.ai-primer.com (third-party) · News digest dated 2026-07-05, seen 2026-10-10. It attributes 16.1% to a social-media summary of the CAIS post.
We show 15.8.
RoboChallenge Table30 · open · opened 24 Sep 2026
The RoboChallenge report (October 2025) gives the task-specific π0.5 baseline an average success rate of 43.7% on Table30. The live v1 leaderboard shows the same baseline (pi0.5, rc_baseline) at 42.67%. A task-by-task comparison (2026-10-10) finds the gap in two of the 30 tasks: arrange fruits in basket is 80% in the report and 40% on the board (score 82.8 against 70.5), and sort electronic products is 40% in the report and 50% on the board (progress score 68.6 in the report, 68.625 on the board). The other 28 tasks match. The net change of 30 points over 30 tasks is exactly the 1-point gap. All 30 per-task entries carry a 2026-05-28 update time. A re-run or a copy error in the report could explain it; the maintainers have not explained either difference.
- 43.7 — arxiv.org (primary) · Figure 9 of the report, "average" row, Pi05 SR column.
- 42.67 — robochallenge.ai (primary) · success_ratio 0.4267 for display_name pi0.5, user rc_baseline, read 2026-09-24 and again 2026-10-10. Per-task values are in leaderboard_task_all.json.
We show 43.7.
SOP-Bench · open · opened 24 Sep 2026
The SOP-Bench paper gives two values for its best result, Claude 4 Opus with the ReAct agent. The abstract and Table 6 say 72.4% task success rate; the text of section 5.1.2 and the conclusion of the KDD camera-ready say 73.8%. The README's approximate "ReAct agents: ~55% average" is an average over all models, not the best. Checked again 2026-10-10: the per-procedure TSR values in Table 4 average 0.725, which matches 72.4%, and their conditional success rates average 0.727, so neither column gives 73.8%. The arXiv v2 conclusion pairs 73.8% with "Claude 4.5 family: 58.8%", where the KDD camera-ready says 63.3%, so the conclusion's numbers changed between versions without the 73.8% changing. The cause is not stated; a question to the authors would settle it.
- 72.4 — arxiv.org (primary) · Abstract and Table 6 (0.724), arXiv v2 and the KDD 2026 camera-ready PDF.
- 73.8 — www.amazon.science (primary) · Section 5.1.2 text, "ReAct performs best with Claude 4 Opus as well (73.8% TSR)", and the conclusion, "Claude 4 family: 73.8% vs. Claude 4.5 family: 63.3% average TSR on ReAct", same PDF. arXiv v2 has the same section 5.1.2 sentence.
We show 72.4.
SpreadsheetBench 2 · open · opened 24 Sep 2026
The SpreadsheetBench site shows two different top scores for V2. The static "Top Score (OVERALL)" box on the V2 overview reads 34.89%, the best scaffolded model in the paper. The leaderboard data file that the page loads listed WPS AI at 50.86% (verified, 2026-08-29) on 2026-09-24 and arito at 54.51% (verified, 2026-09-13) on 2026-10-10. The box looks stale: it has not changed while the file did. We show the leaderboard value.
- 54.51 — spreadsheetbench.github.io (primary) · Leaderboard data file, seen 2026-10-10. arito, marked verified, dated 2026-09-13.
- 50.86 — spreadsheetbench.github.io (primary) · Leaderboard data file, seen 2026-09-24 (WPS AI, then rank 1). Entry is marked verified. On 2026-10-10 it is rank 2.
- 34.89 — spreadsheetbench.github.io (primary) · Static statistics box on the V2 overview page, seen 2026-09-24 and again 2026-10-10. Matches the Claude Opus 4.6 (SWE-agent) row and the paper abstract.
We show 54.51.
SpreadsheetBench · open · opened 23 Sep 2026
The SpreadsheetBench site shows two different top scores for the 912-question V1 set. The static "Top Score (OVERALL)" box reads 70.48%. The leaderboard data file that the page loads lists Qingqiu Agent at 83.11% (verified, 2026-06-23). The box looks stale. We show the leaderboard value.
- 83.11 — spreadsheetbench.github.io (primary) · Leaderboard data file, seen 2026-09-23 and again 2026-10-10. Entry is marked verified.
- 70.48 — spreadsheetbench.github.io (primary) · Static statistics box on the V1 overview page, seen 2026-09-23 and again 2026-10-10. Matches the Gemini in Google Sheets row dated 2026-03-10.
We show 83.11.
SWE-Bench Pro · resolved · opened 23 Sep 2026
Scale's public leaderboard shows a best resolve rate of 61.5% (Muse Spark 1.1, mini-swe-agent, added 2026-07-09). OpenAI's 2026-07-08 audit post says frontier models reached 80.3% on the same 731-task public split, without naming the model or harness. We show the leaderboard value.
- 61.5 — labs.scale.com (primary) · Scale AI public leaderboard, Muse Spark 1.1 with mini-swe-agent, 61.50 ± 3.10, seen 2026-09-23 and 2026-10-10.
- 80.3 — openai.com (lab-reported) · OpenAI post dated 2026-07-08. "Frontier models improved from a pass rate of 23.3% to 80.3% in eight months." No model or harness named.
- 80.3 — www.anthropic.com (lab-reported) · Anthropic's Claude Fable 5 announcement (2026-06-09), benchmark table, row "Agentic coding, SWE-Bench Pro", column "Claude Mythos 5 / Fable 5". The table shows the higher score of the two models. Seen 2026-10-10.
We show 61.5. Resolution: Found 2026-10-10: the 80.3% equals the SWE-Bench Pro score that Anthropic reported for Claude Mythos 5 / Fable 5 in its 2026-06-09 announcement, one month before OpenAI's post. OpenAI does not cite its source, but the value and the timing match. It is a lab-reported run with the lab's own setup, and Scale's board has no row for either model, so the two numbers measure different runs. The atlas keeps the board value; the V1 board is now retired.
CyberBench v1.1 · open · opened 24 Sep 2026
The CyberBench v1.1 page's Key Takeaways text and its leaderboard data name different PoC track leaders, and the gap has persisted across updates. On 2026-09-24 the text said Muse Spark 1.3 Max (63.3%) while the data had MiMo V2.6 Flash at 65. On 2026-10-10 the text says Kimi K3 (68.3%), with GPT-6 Sol close behind (66.7%), while the data (updated 2026-10-07) lists Gemini 4 Argon at 70 ahead of Kimi K3 at 68.333. The likely cause is that the hand-written text lags the data when models are added; Vals has not said so. We show the table values; the PoC track is not our headline metric.
- PoC track led by Muse Spark 1.3 Max, 63.3% — www.vals.ai (primary) · Key Takeaways text on the page, read 2026-09-24. Replaced by 2026-10-10.
- PoC track led by Kimi K3, 68.3%; GPT-6 Sol 66.7% — www.vals.ai (primary) · Key Takeaways text on the page, read 2026-10-10.
- PoC track led by Gemini 4 Argon, 70; Kimi K3 68.333; GPT-6 Sol 66.667 — www.vals.ai (primary) · Leaderboard data embedded in the page HTML ("poc" section), updated 2026-10-07, read 2026-10-10. On 2026-09-24 the same data listed MiMo V2.6 Flash first at 65.
We show Gemini 4 Argon 70 (PoC track, in the results notes).
Reference data
- O*NET 31.0 Database (U.S. Department of Labor, Employment and Training Administration): occupations, job titles, and tasks. CC BY 4.0. onetcenter.org
- BLS OEWS, May 2025 national estimates (U.S. Bureau of Labor Statistics): employment and wages. Public domain. bls.gov/oes