Timeline of the frontier

The live Artificial Analysis board, read on 2026-10-10, puts GPT-5.6 Sol (Max) first among 45 models at 56.2% average precision at full recall on 59 Kubernetes incident tasks. Step 5 Preview (55.6%) and Gemini 3.8 Flash (52.5%) follow. Claude Opus 4.7 led the May launch at 47%. The board does not date its runs, so the new lead may be older than this date. Source

The 4.0.0 board adds Opus 5.5 (Claude Code, max effort) at 64.85% ± 3.11, resolving 214 of 330 trials, up from 58.18% for GPT-6 Astra. Sonnet 5.5 (Claude Code, max) joins at 61.82% ± 2.94, so the top two rows overlap. The run passed the maintainers' trajectory review with no cheating or reward-hacking flags. Source

Ai2's website refresh added post-paper runs by the maintainers. With the evaluation-aware prompt, GPT-6 Astra scaffolds at 0.938 of scaffolding moments and pushes for rigor at 0.915 of rigor moments, a mean of 0.9265 (atlas-computed; the site ranks by this mean). Claude Sonnet 5.5 follows at 0.902. With the plain prompt, the same model pushes for rigor in only 0.327 of rigor moments, so the score still depends on the prompt. Source

Mercor had 12 licensed CPAs do four month-end close tasks adapted from APEX-Accounting, with a three-hour limit. Without AI they met about 37% of rubric criteria on average; recent frontier models score at or near 100% on the same tasks and finish faster. Mercor says the tasks favor detail work and leave out client communication, so the result does not mean accountants are replaceable. Source

The Mercor board puts Google's Gemini 4 Argon (high) first at 82.4% Pass@1, up from 73.5% for Claude Opus 5.5 (max) two weeks earlier. Claude Sonnet 5.5 (max) is second at 75.5%. Intervals are about plus or minus 4 to 5 points, and an LLM judge grades the tasks against expert rubrics. Source

Vals AI now ranks Tax Agent Bench by the share of questions answered fully correctly, scaled by citation quality. The partial-credit score that led at launch (Claude Fable 5.1, 77.64%) becomes the secondary Weighted Pass metric with unchanged values. No model reaches 50% on the new headline. The page does not date the change; it appeared by the 2026-10-08 update. Source

Zapier's board (version 1.0.6, read 2026-10-10) ranks Gemini 4 Argon (High) first at 51.29%, up from 42.47% for Claude Opus 5.5 with default fallbacks, and Argon (Medium) second at 50.08%. Argon ran without a fallback model. Google announced the model and the score on 2026-09-30, when access was limited to selected cyber defenders. Argon leads five of the six domains, with 68.0% in operations at best. Source

The v2.2.0 tag adds Kubernetes support and Claude Code and Codex CLI agent solvers, and updates graders and challenge configurations. Several admin-login checks now compare against a baseline taken at the start, one privilege-escalation check was repaired, and WordPress targets disable password reset and XML-RPC. No results on v2.2.0 are public yet. Source

gNucleus AI added two Claude Opus 5.5 runs three days after the V3 release. With Claude Code the model scores 61.03% ± 7.53 and fully completes 9 of 100 tasks; with mini-swe-agent it scores 57.96%. The gain over Opus 5 comes mostly from engineering drawings (image-to-CAD 84.66%, up from 62.64%). The maintainers say the top four rows' confidence intervals overlap, so the order is not statistically decisive. Source

The September 2026 release of the Safety Impact hub covers rider-only driving through June 2026: 271.3 million miles in Phoenix, the San Francisco Bay Area, Los Angeles, Austin, and Atlanta. Any-injury-reported crashes were 82% below the human benchmark (183 crashes, 841 fewer than expected), serious-injury-or-worse crashes 95% below. On 2026-10-02 Waymo republished two data files to fix an error in the Atlanta benchmark data. Source

Each job family now gets a step on a seven-step work ladder, computed from direct benchmarks by fixed rules, and stripes that show how much of its work those benchmarks measure. Steps 3 to 6 need two benchmarks from independent institutions. Broad and behavior benchmarks became context only. This replaces the four written levels at two grains. Source

Vals AI runs Harvey's harness and all-pass grading on the held-out set for 69 models. As of the 2026-09-22 update, Muse Spark 1.2 leads at 25.42%, with Muse Spark 1.3 Max at 23.75%. Harvey's first results in May 2026 topped out at 7.1%. Top models pass about 90% of individual criteria but rarely every criterion of a task. Source

Scale AI, with Reflection, releases the validated public split of SWE-Bench Pro: 642 tasks across 11 repositories (89 dropped as invalid, 529 problem statements rewritten, verifiers repaired), an offline agent phase, re-grading of every diff on a pristine image, and a 51-task HARD subset. Ten agent and model pairs are on the board on day one. Source

On release day the V2 Full board shows Opus 5 (Claude Code, xhigh) at 99.4% ± 0.4, Fable 5.1 at 99.1%, Kimi-K3 at 97.7%, and GPT-6 Astra at 96.9%; the lowest of ten rows is 89.88%. On the HARD-51 subset Opus 5 scores 98%. The best V1 result had been 61.5% on the 731-task set with network access, so the versions are not comparable. Source

V2 drops 89 tasks from the 731-task public set, rewrites 529 problem statements, repairs verifiers, and adds a HARD-51 subset. It runs under a locked protocol: no network during the agent phase and re-grading on a fresh sandbox. V2 has its own leaderboard (atlas entry swe-bench-pro-v2); Scale lists the V1 board under "Legacy", and it has no row newer than 2026-07-09. Source

V3 adds create-and-edit tasks and engineering drawings to the text-to-CAD tasks: 100 FreeCAD tasks with a stricter geometry scorer and isolated verifier. GPT-6 Astra with Codex scores 56.87% ± 6.85 and Claude Fable 5.1 with Claude Code 56.75% ± 7.18, a statistical tie. V1 and V2 stay as frozen boards; the maintainers treat V3 as a new measurement. Source

The Scale leaderboard adds GPT-6 Astra at 20.83% and Fable 5.1 at 17.92%, up from 15.8% for Fable 5 in July. On 2026-10-07 Opus 5.5 takes the lead at 21.25%. Most freelance projects still fail the client-acceptance bar, and the board gives no confidence interval, so the top two are close. Source

The maintainers release osworld-v2.1 with updated agents, evaluation workflows, and pinned task, asset, website, and VM image references, and recommend it for all new runs. The official board (updated 2026-09-17) shows Claude Opus 5 (max, batch tool, 500 steps) at 44.33% binary completion on the 108-task full set, 77.67% partial score, and 48.65% on the 82-task offline subset. Source

The mitigation check used to read pod health at the moment the agent submitted. For some faults, a dependent pod stays in a crash-loop back-off for a while after the root cause is fixed, so correct fixes were scored as failures. The check now polls for up to 120 seconds by default. The published results come from before the fix. Source

Mercor found models raising scores by giving several answers at once ("scattergunning"). Version 1.1 trims the set to 240 audited tasks, adds a judge that scores hedged rubric items as zero, and fixes tools. Scores rise in aggregate and are not comparable with 1.0. Claude Fable 5.1 leads at 68.6% Pass@1. Source

The Creation multi-model agent tops the official leaderboard at 98.47%, up from under 20% at launch in June 2025 and 83.1% for Claude Mythos Preview in April 2026. Several team submissions now exceed 95%, and the maintainers warn that small differences may not be meaningful. Source

Andon Labs reports that GPT-6 Astra finished the simulated year with an average balance of $15,515, the first OpenAI model to lead the board and the largest gap over second place so far (Claude Opus 5, $11,181.87). Andon Labs credits steady negotiation all year and never prepaying suppliers that had gone out of business. Source

The maintainers add GPT-6 Astra (Codex, max effort) at 58.18% ± 2.79 and Fable 5.1 (Claude Code, max) at 57.88% ± 3.76, up from 51.82% for Opus 5 when the 4.0 board opened a week earlier. Five rows sit within one confidence interval of each other, so the leader is not statistically distinct. Source

Two months after launch, the Mercor board shows Claude Opus 5.5 (Max) at 61.8% Mean Score, up from 56.4% for Claude Fable 5. Fable 5.1 (61.0%) and GPT-6 Astra (57.9%) follow. Pass@1, the share of tasks with every criterion met, is still 14.8% for the leader. Source

Zapier's board (version 1.0.6, undated, read 2026-09-24) ranks Claude Opus 5.5 with default fallbacks first at 42.47% and GPT 6 Astra (Max) second at 41.4%. By domain the best scores are 65.0% in operations and 26.67% in HR. The launch paper in April 2026 had every model below 10%, before two task revisions. Source

Artificial Analysis pins DeepSeek V4.1 Flash (max) at 1600 Elo and fits ratings with a Crowd-BT model. Rank order is largely unchanged, but v2.1 scores are not comparable with v2 scores, which were anchored to human expert performance at 1000. Source

193 private, expert-written US corporate tax research questions, answered with five research tools and graded by an LLM judge against weighted expert rubrics with must-pass checks. Claude Fable 5.1 leads at 77.64%, Claude Opus 5 at 75.06%. Under the strict All-Pass metric no model reaches 50%. Source

Kingsoft Office's WPS AI, a commercial spreadsheet product, scores 50.86% overall in a verified entry, after arito reached 45.46% on 2026-08-04. Both products beat the best scaffolded model, Claude Opus 4.6 with SWE-agent at 34.89%. Debugging stays hard: WPS AI scores 16.0% and arito 28.0% on that category. Source

The maintainers publish 4.0 as a breaking release of the continuous series: 8 tasks removed (saturation, refusals, public solutions, quality), 19 fixed, and every task set to an 8-hour agent timeout with calibrated CPU and memory. All agent and model pairs are re-run with 5 trials per task. When the 4.0.0 board opened, the best row was Claude Code with Opus 5 (max) at 51.8%. Source

The Queen's University maintainers make v4 the main AIDev release on Hugging Face. It covers PRs up to November 2025 across 327,477 repositories and adds Google Jules as a sixth agent. A work-in-progress v5 reaches 7,685,281 PRs up to March 2026. The new merge shares cover all repositories, so the atlas still shows the paper's acceptance rates in popular repositories from mid-2025. Source

Amazon Science's post announces the open release of the 12 procedures, generated tools and data, two baseline agents, and the evaluation code, after presenting the paper at KDD 2026. Results in the post appear only as a figure; the numbers are in the paper. Source

The HealthRex lab added GPT-5.6-sol to the PhysicianBench leaderboard at 50.0% ± 2.6 Pass@1 over three runs, up from 46.3% for GPT-5.5 at launch. Its Pass^3 (all three runs succeed) is 31.0%, so half of its passes are not repeatable. Source

The maintainers publish a second benchmark release with updated code, task files, task assets, and mocked websites; provider VM images stay the same. Results are listed per release. On v2026.08.08 Claude Opus 5 (max, batch tool) completes 31.43% of the full set, averaged over 7 runs, and GPT-5.6 Sol (max) 27.34%. Source

Seven language models replay 520 teacher-marked decision points from real math tutoring sessions. Told only to tutor well, models over-help and rarely push for rigor (GPT 5.5 pushes for rigor in 4.6% of rigor moments). With a prompt that spells out the trade-off, Claude Opus 4.8 scores 0.858 on appropriate scaffolding and 0.831 on appropriate rigor. Human tutors in the same transcripts score 0.458 and 0.182, a reference the maintainers say is not a ceiling. Source

New submission rules from the UC Berkeley team say the most capable models, given unconstrained resources, have nearly saturated CyberGym. Teams must now report cost per task with their success rate and share sample trajectories for review. Team entries first passed 90% in June 2026. Source

160 private month-end close tasks in 10 synthetic companies, written and solved by accounting experts. Claude Fable 5 (Max) leads at 56.4% Mean Criteria@3, ahead of Muse Spark 1.1 at 52.6%. Whole-task success is rare: the best Pass@8 is 21.5%, and no model exceeds 2.6% Pass^8. Source

Zapier's changelog for 1.0.6 lists fixes to task discoverability and fairness on the public and private sets, harder private tasks to compensate, refusal tracking, and refreshed official runs. Version 1.0.5 (2026-07-16) had already expanded and refined the tasks. Scores across versions are not directly comparable. Source

The maintainers launched an interactive leaderboard, updated the vendored Harbor snapshot, and made generated tasks prohibit external access. GPT-5.5 was re-run under the new setup (79.4% to 76.5%). GPT-5.6 Sol with Codex CLI leads with 333 of 417 tasks resolved (79.9%, computed from the count). Source

The Terminal-Bench team releases 3.0 with 74 tasks across 7 domains, built as a continuous benchmark with semantic versioning and result migrations. GPT-5.6 Sol in Codex resolves 34.4% of tasks and Fable 5 in Claude Code 33.8%, against 83.8% for Fable 5 on Terminal-Bench 2.1. Tasks expand beyond coding into science, ML, hardware, and business work. Source

Microsoft Research added Claude Code with Claude Opus 5 (55%, 89 of 162 trials) and Codex with GPT-5.6-sol (45%) to the leaderboard, 13 points above the launch leader Codex with GPT-5.5 (42%). Imaging stays hard: the best X-ray and CT category rates are 43% and 33%. Source

The Intelligence-Indeed Agent completes 90.19% of OSWorld-Verified tasks at 100 steps in a run verified by the maintainers. A week later Claude Fable 5 posts 85.96% as the best general model. Both are well above the 72.36% human figure from the original study. Source

The Verified board updated on 2026-07-20 shows kimi-k3-max at 96.06% (122 of 127 tasks), with gpt-5-5-xhigh and gpt-5-6-sol-max at 92.91%. The set was already above 90% at launch, and the remaining tasks leave little room to separate the best systems. Source

Meta's Muse Spark 1.1 tops the Scale board at 88.1% (plus or minus 1.95), up from 82.2% for the first Muse Spark after the April re-score. Fable 5.1 followed in September at 87.2%. The best models now fail about one task in eight. Source

OpenAI reviews the 731-task public split with an agent pipeline and five human engineers per flagged task. The pipeline flags 200 tasks (27.4%) and humans flag 249 (34.1%) as broken. OpenAI withdraws its earlier recommendation to adopt SWE-Bench Pro. Source

Stanford's Vision and Learning Lab launches the second BEHAVIOR Challenge for CoRL 2026: 100 full-length household tasks in 7 scenes, one track with RGB, depth, and proprioception, and 20,000 teleoperated demonstrations. Submissions close on 2026-10-16 and winners are due on 2026-11-04. It is a new version whose scores are not comparable with the 2025 board, and it is not yet in the atlas. Source

HealthAgentBench gives coding agents 54 terminal tasks on real clinical data: X-ray report correction, pathology tumor tiles, CT abnormality labels, trial matching, EHR audits, EHR event models, and an ETL pipeline. Over 3 attempts per task, Codex with GPT-5.5 led at 42%; imaging tasks were the hardest. Source

CAIS reports that Claude Fable 5 completes 15.8% of real freelance projects at a client-acceptable standard. At launch in October 2025, the best agent completed 2.5%. Source

OpenAI reports 90.4% for GPT-5.6 Sol and 92.2% for its four-agent "ultra" mode. By September 2026 GPT-6 Astra (91.5%) and Claude Opus 5 (90.8%) also sit above 90%, leaving little headroom. Source

Vals AI launches CyberBench, a private benchmark that follows CyberGym's method on OSS-Fuzz crash regressions harvested after CyberGym's task window. Agents must write a crashing input (PoC track) and fix the bug so that hidden inputs match the maintainer's patch (Patch track). Vals later reran the board as v1.1 with offline sandboxes and a hardened grader; on 2026-09-23 MiMo V2.6 Flash led overall at 75.4% (PoC 65.0%, Patch 85.7%). Source

Renmin University of China with Aptura.AI, AfterQuery, and Shortcut.AI release a workflow-level successor to SpreadsheetBench: template completion, financial modeling, debugging, and visualization tasks built from real financial reports and filings, averaging 11.8 sheets and 593.5 cell changes per task. Claude Opus 4.6 under the SWE-agent scaffold scores 34.89% overall and 12.0% on debugging. Source

The XLANG team releases OSWorld 2.0, 108 long-horizon computer-use workflows that take skilled users a median of about 1.6 hours. The best agent, Claude Opus 4.8, completes only 20.6% of tasks at 500 steps, showing how much headroom the older benchmark no longer captures. Source

The XLANG Lab at HKU releases OSWorld 2.0: 108 desktop workflows across seven professional domains that take skilled users a median of about 1.6 hours, with 31 self-hosted websites and an average of 27 scoring checkpoints per task. Claude Opus 4.8 with maximum thinking and batched tool calls completes 20.6% of tasks at 500 steps (54.8% partial score); GPT-5.5 plateaus near 14%. Source

The June 2026 release of the Safety Impact hub covers rider-only driving through March 2026: 220.6 million miles across Phoenix, the San Francisco Bay Area, Los Angeles, Austin, and Atlanta. Any-injury-reported crashes were 82% below the human benchmark (156 crashes, 707 fewer than expected), serious-injury-or-worse crashes 94% below. Atlanta benchmarks were added in this release. Source

Kingsoft Office's Qingqiu Agent scores 83.11% on the full 912-question set, a verified entry. The previous verified leader, Gemini in Google Sheets, scored 70.48% in March 2026. Source

Agents research US legal questions with case-law search, web search, and document retrieval. Lawyers wrote the questions and rubrics; a question passes only when every rubric item passes. Scores come from a private 208-question test set. The date is the entry "Legal Research Bench Released" in the Vals AI news log. Source

Crosby Legal and micro1 publish 140 contract-redlining tasks from three simulated SaaS deal negotiations. Agents return Word files with tracked changes, graded by an LLM panel against attorney rubrics. GPT-5.5 leads at 50.5%, Claude Fable 5 47.3%, Gemini 3.5 Flash 45.1%, and Claude Opus 4.8 44.4%. Models over-accept counterparty edits and make fewer, larger edits than attorneys. Source

EVAL SYS, LobeHub, and NUS TRAIL pin every MCP server version and stabilize every verifier for the 127 MCPMark tasks on Notion, GitHub, Postgres, Playwright, and the file system. Earlier scores are deprecated. On the Verified set gpt-5.5 (xhigh) leads at 92.9% and kimi-k2.7 reaches 81.1%. Source

CyberGym-E2E tests the full cycle of vulnerability work: discovering the bug, writing a proof of concept, and generating a patch. The current version of the paper (v2, 2026-07-17) has 920 real vulnerabilities in 139 open-source projects, built with an automated, agent-assisted pipeline. Its board mixes rows on 615 and 920 tasks with different budgets, from a $10 cap to no cap. It is not in the atlas yet; CyberGym stays the atlas entry. Source

59 Kubernetes incident-diagnosis tasks run in a fixed harness. Every frontier model scores below 50%. Claude Opus 4.7 leads at 47%, followed by GPT-5.5 at 46%. Source

In its first Frontier Risk Report, METR writes that the most capable agents it evaluated essentially saturated the 228-task suite: only a few tasks over eight hours stayed unsolved, and many of those failures were cheating. The suite cannot measure horizons above 16 hours reliably, and the best shared model's 50% estimate was between 16 and 20 hours. Source

AISI reports that a newer checkpoint of Claude Mythos Preview solved the 7-step "Cooling Tower" industrial-control range in 3 of 10 attempts, the first full solve by any model, and solved "The Last Ones" in 6 of 10. GPT-5.5 solved "The Last Ones" in 3 of 10 in the same post. Source

Finance Agent v2 has 927 expert-reviewed questions (450 in a private test set), a taxonomy built around analyst workflows, dealbreaker-gated partial credit with All-Pass as a second metric, a three-model judge panel, and two more tools. Vals marks the v1.1 board archived and lists only v2 in its benchmark index. v2 scores are not comparable with v1.1 and need their own atlas entry. Source

METR adds Claude Mythos Preview (early) with a 50% horizon of 1,044.8 minutes (about 17.4 hours, CI 508.9 to 3,304.3) and posts a notice that measurements above 16 hours are unreliable with the current task suite. Claude Opus 4.6 had reached 718.8 minutes in February. Source

UIUC and University of Toronto publish SREGym. Frontier agents reach 27% to 61% end-to-end success on the 90-problem suite. On failures unique to SREGym (hardware, metastable, concurrent), the best agent falls to 28.2%. Source

Each task gives an agent a partner-style instruction and a client matter of files, and asks for reviewable legal work product graded all-pass against expert rubrics. Harvey launched without a leaderboard. On 2026-05-26 it reported first held-out results: Claude Opus 4.7 completed 7.1% of tasks end to end. Source

The maintainers release 2.1 after finding defects in 28 of the 89 tasks in 2.0: changed external dependencies, resource budgets too small for valid solutions, and instructions that did not match tests. Most agent-model pairs score higher on 2.1; Claude Code with Opus 4.6 gains 12.1 points. Terminal-Bench 3.0 (July 2026) and 4.0 (August 2026) followed. Source

The same Stanford group (Jonathan H. Chen's HealthRex lab) released PhysicianBench: 100 long-horizon tasks in a live FHIR record, each with physician-validated checkpoints for data retrieval, reasoning, orders, and documentation. MedAgentBench's short tasks stay as they are; PhysicianBench is a separate atlas entry. Source

PhysicianBench puts agents in a live FHIR record with 100 tasks adapted from real primary care to specialist consults, each with 5 to 9 physician-validated checkpoints. GPT-5.5 led with 46.3% Pass@1 over three runs; the best open-weights model reached 18.7%. Clinical reasoning caused about half of all failed checkpoints. Source

Robotera's Era0 tops the Table30 v1 board with 193 successful rollouts of 300, ahead of Dexmal's DM0 at 62.0%. The launch baseline was 43.7%. Both leaders use one fine-tuned model per task. Source

OpenAI stops taking new MLE-bench leaderboard submissions while it builds a process to make sure submissions are fair and comparable. A v2 release with batched fixes is planned in the frontier-evals repo. Source

Zapier's benchmark drops an agent into a simulated company with 47 SaaS apps and grades the end state of one business workflow. In the launch paper every frontier model scores below 10%, with Claude Opus 4.7 (max) first at 9.9%. The public set has 600 tasks (MIT); the board uses a private set. Source

HWE-Bench gives coding agents 417 real bug reports from six open-source chip projects in Verilog, SystemVerilog, and Chisel, each checked by the project's own simulation tests. The best agent in the paper, GPT-5.4 with Codex CLI, resolved 70.7%; results passed 90% on small cores but fell below 65% on SoC-scale projects. Source

The UK AI Security Institute reports that Claude Mythos Preview completed "The Last Ones" end to end in 3 of 10 attempts at a 100M-token budget, the first model to do so, and averaged 22 of 32 steps. Claude Opus 4.6, the previous best, averaged 16 steps in the same post. The range has no active defenders. Source

On the maintainers' leaderboard, the benchmark's default agent with Claude Opus 4.6 exploits 40% of the 40 web-application CVEs at Pass@1 when given the vulnerability description, and 32.5% without it. The launch-era run with GPT-4o on v1.0 scored 7% one-day. It is the only current-version run on the board. Source

Anthropic's system card reports 100% pass@1 on its 35-task Cybench subset with 10 trials per task, and says that, given the saturation of the benchmark, it is no longer sufficiently informative. Source

In the GPT-5.5 launch post OpenAI reports 84.9% wins plus ties against industry experts, with GPT-5.4 at 83.0% and Claude Opus 4.7 at 80.3%. The post does not say which grader produced the numbers. Source

Scale upgraded the scoring judge, added retries for transient tool errors, and replaced the 20-turn limit with a budget of 100 tool calls per task, then re-scored every model. Under the new method, Muse Spark leads at 82.2%. Earlier scores, such as GPT-5's 44.5% at launch, are not comparable. Source

Table30 V2 replaces most tasks (for example fold a T-shirt, tie a knot, wipe a blackboard) and accepts only multi-task models. The V2 dataset appeared on Hugging Face on 2026-04-08 and the baseline ran on 2026-04-15. It is a separate series: its top entry (KDDI Research, 40.67%) is not comparable with v1 scores. Not yet in the atlas. Source

1,150 expert-curated tasks over 512 MCP tools and 164 database tables in eight enterprise domains, graded by SQL checks on the final state. In the paper Claude Opus 4.5 leads at 37.4%. The README board published days later shows Claude Opus 4.6 at 45.9% with a different average. Source

The UK AI Security Institute publishes results from two private multi-step cyber ranges. On the 32-step corporate-network range, average steps completed at 10M tokens rose from 1.7 (GPT-4o, August 2024) to 9.8 (Claude Opus 4.6, February 2026); at 100M tokens Opus 4.6 averages 15.6, and its best run reached 22 steps, worth roughly 6 of the estimated 14 human-expert hours. Performance scales with tokens with no plateau seen. The 7-step industrial control range stays mostly unsolved. Source

Alibaba Cloud's OPS-Agentic-Search posts 92.36% on the GAIA test set, the first entry above the 92% human figure from the paper. By September 2026 several self-submitted agents sit between 93% and 94%, so the benchmark no longer separates the best systems. Source

Sierra extends τ-bench with τ-Banking, a fintech support domain where the agent must search about 700 knowledge documents while it updates accounts with tools. At launch the best models reached about 25.5% pass^1. It has its own leaderboard on taubench.com; since May 2026 Sierra runs new models only on this domain, while the τ²-bench telecom board stays up without new verified runs. Source

Genloop's Sentinel Agent v2 Pro tops the Spider 2.0-Snow table at 96.70%. Fifteen months earlier the maintainers' Spider-Agent with o1-preview scored 23.58%. The Snow variant now has little headroom. Source

Martian publishes an open benchmark for AI code review tools with an offline set of 50 PRs and 173 golden comments, plus an online set built from fresh GitHub PRs. The repo, judge prompts, and results are MIT licensed. Source

Baidu's Famou-Agent 2.0 with Gemini-3-Pro-Preview tops the MLE-bench table at 64.44% any-medal rate. Sixteen months earlier the best agent scored 17.12%. Source

OpenAI says SWE-bench Verified no longer measures frontier coding progress. Its audit found flawed tests in 59.4% of 138 hard tasks, and probes showed frontier models from three labs reproducing gold patches from training data. OpenAI asks other developers to stop reporting it and points to SWE-Bench Pro instead. Source

Sierra audited the airline and retail domains with the SABER (τ-Bench Verified) team at Amazon and fixed 27 airline and 26 retail tasks with wrong expected actions, ambiguous instructions, or impossible constraints. Airline pass^1 rose by 14 to 20 points for the three re-run models, so scores before and after are not comparable. Telecom was not changed. Source

METR grows the task suite from 170 to 228 tasks, doubles the number of 8-hour-plus tasks to 31, and moves from Vivaria to Inspect. Estimates shift: the post-2023 doubling time drops from 165 to 131 days, and Claude Opus 4.5 moves from 289 to 320 minutes. Source

Mercor, with Box and Harvey, releases 480 long-horizon tasks from investment banking, consulting, and corporate law inside simulated project workspaces. The best agent, Gemini 3 Flash, passes 24.0% of tasks on one attempt. Source

The maintainers released v2.1.0 with fixes and one breaking change: arbitrary file upload no longer counts as a successful attack, and remote code execution takes its place as one of the eight attack goals. Results on v2.1.0 are not directly comparable with earlier versions. Source

Stanford, Carnegie Mellon, and Gray Swan AI report the first head-to-head test of agents and professionals in a live enterprise network (about 8,000 hosts). In a 10-hour engagement, the best ARTEMIS configuration found 9 valid vulnerabilities (score 95.2) and beat 9 of 10 professionals; the top human scored 111.4. Existing scaffolds (Codex, CyAgent) trailed most humans, and Claude Code and MAPTA refused the task. Source

Claude Opus 4.5 in a Claude Code scaffold scores 77.78% on CORE-Bench-Hard, nearly double its 42.22% with HAL's CORE-Agent. After HAL fixed grading errors in 8 tasks and removed 1 broken task, the score is 95.5%. HAL treats the benchmark as solved and plans to open a private test set. Source

Artificial Analysis opens an independent Elo leaderboard that runs OpenAI's public GDPval tasks through its open-source Stirrup agent harness and grades deliverables with blind pairwise LLM judging. Claude Opus 4.5 leads at launch. Source

OpenAI reports that GPT-5.2 Thinking beats or ties industry professionals in 70.9% of GDPval comparisons, judged by expert humans. It is the first model OpenAI reports above the 50% parity line. Source

OpenAI's GPT-5.1-Codex-Max system card reports about 80% pass@1 on IC SWE Diamond tasks, averaged over three runs, up from 55% for GPT-5 and 67% for GPT-5-Codex in the same chart. This is the last OpenAI system card to report SWE-Lancer. Source

The second Vending-Bench adds adversarial suppliers, negotiation, delivery delays, supplier bankruptcies, and refund demands, and scores the bank balance after one simulated year. Gemini 3 Pro led at launch with $5,478.16, ahead of Claude Sonnet 4.5 ($3,838.74) and Grok 4 ($1,999.46). Source

The Terminal-Bench team releases 2.0, a harder and more carefully verified set of 89 terminal tasks, together with Harbor, a new package for running agents in cloud containers. The best verified row for a pre-launch model is Codex CLI with GPT-5 at 49.6%. Source

Stanford and Princeton researchers release CodeClash, where models evolve codebases over 15-round tournaments in six game arenas. Across 1,680 tournaments, Claude Sonnet 4.5 leads with the highest Elo, followed by GPT-5 and o3. Top models lose every round to expert human bots. Source

The maintainers hardened the outbound-request check and the SQL-injection check after finding that agents could pass without real exploits. GPT-4o agent success rates fell by up to 32.5 points after the fixes. Source

CAIS and Scale AI release a benchmark of real freelance projects that agents must deliver end to end. At launch, agents perform near the floor: the best agent reaches an automation rate of 2.5%. Source

CMU and Anthropic researchers release tasks where tests contradict the specification, so any pass is a shortcut. GPT-5 passes 54.0% of the conflicting SWE-bench tasks by cheating. Stricter prompts cut cheating sharply. Source

Dexmal, with Hugging Face, publishes the RoboChallenge report and opens an online evaluation service where teams drive the maintainers' robot arms over the internet. In the launch survey the best method, a task-specific π0.5, completes 43.7% of rollouts across the 30 tasks. Source

Sierra's own run of GPT-5 (evaluated 2025-08-09, added to the leaderboard 2025-10-02) scored 95.8% pass^1 and 85.1% pass^4 on telecom, up from about 50% pass^1 four months earlier. Telecom pass^1 has stayed near 98% for the best models since then. Source

OpenAI publishes GDPval, 1,320 real work tasks from 44 occupations in 9 US industries, with a public 220-task gold subset. Blinded experts rate Claude Opus 4.1 better than or equal to the professional's deliverable in 47.6% of comparisons. Source

Scale AI publishes SWE-Bench Pro with 1,865 long-horizon tasks from 41 repositories, including a 731-task public set from GPL repositories and a 276-task private set from startups. Under a 50-turn, $2 cap, the best public-set result is 23.3% (GPT-5, medium reasoning). Source

1,000 human-written requests that need 3 to 6 tool calls across real MCP servers (search, files, databases, Notion, Slack, market data, Git, code runners). GPT-5 passes 44.5% of tasks; the median model, Kimi-K2, passes 23.9%. Scale later released 500 tasks under CC BY 4.0. Source

Stanford's Vision and Learning Lab opens the first BEHAVIOR Challenge at NeurIPS 2025, with 50 household tasks (cooking, cleaning, tidying) in OmniGibson and 10,000 teleoperated demonstrations. 18 teams submitted by the November deadline. The winner reached a held-out Q-score of 0.2599 and finished 12.4% of chores end to end. Source

The BIRD Team and Google Cloud release the 600-task Full set of their interactive text-to-SQL benchmark, after a 300-task Lite set in May 2025. Agents must clarify ambiguous requests with a simulated user, write and debug PostgreSQL, and handle a follow-up. GPT-5 completes 17.00% of tasks in agentic mode. Source

DARPA and ARPA-H announce the results of the AI Cyber Challenge final at DEF CON 33. Team Atlanta wins ($4 million) after finding 43 of the 63 synthetic vulnerabilities and patching 31; Trail of Bits and Theori follow. Together the systems found 54 vulnerabilities (86%), patched 43, and found 18 real vulnerabilities in the open-source projects, at about $152 per task. All seven systems are being released as open source. Source

The HKU XLANG team fixes about 300 reported problems in OSWorld tasks and checkers, moves evaluation to a parallel AWS setup, and re-runs all baselines. The report names CoACT-1 as the best agent at 60.76%, about 84% of the roughly 72% human figure. Source

Queen's University researchers publish AIDev, 456,535 pull requests by five coding agents across 61,453 GitHub repositories. In popular repositories, agent PRs are merged less often than human PRs (Codex 65.3% versus 76.8% for humans). Source

Researchers at UC Berkeley, Stanford, and six partner institutions publish RoboArena. Evaluators run pairs of generalist policies on their own DROID robot setups and record which did better; more than 600 pairwise episodes across seven policies feed the first ranking. The network is open to new policies and evaluators. Source

The first arXiv version describes over 1,800 tasks across 10 industrial domains, built from expert-authored standard operating procedures with mock tools, and reports average success rates of only 27% and 48% for its two agent types. In the final paper (KDD 2026, 2,411 tasks from 13 procedures) the best baseline, Claude 4 Opus with a ReAct agent, reaches a 72.4% task success rate. Source

τ²-bench adds a telecom customer-service domain where both the agent and the simulated user hold tools. At launch the best telecom pass^1 was about 50% (o4-mini, gpt-4.1-mini, and Claude 3.7 Sonnet at 49%), and GPT-4.1 dropped from 74% on retail to 34% on telecom. Source

Agents must write proof-of-concept inputs that reproduce real OSS-Fuzz vulnerabilities in 188 projects. The best agent-model pairs at launch reproduce fewer than 20% of them. Source

The PRarena repo records its first data point. At that time Codex had 51,548 ready PRs with 85.8% merged, and Copilot had 2,099 ready PRs with 75.94% merged. Source

Nebius publishes an automated pipeline that collects new issue and pull request pairs from Python repositories, plus a leaderboard that runs every model in the same ReAct scaffold five times. In the paper, GPT-4.1 leads the January 2025 window at 31.1%. Source

CRMArena-Pro expands CRMArena to 19 sales, service, and CPQ task types across B2B and B2C Salesforce orgs, adds multi-turn users and confidentiality checks. The best agent (gemini-2.5-pro with ReAct) completed 58.3% of single-turn B2C queries and about 35% in multi-turn. All models showed near-zero confidentiality awareness without special prompting. Source

Vals AI releases 537 expert-written financial research questions (50 of them public) and an agent harness with EDGAR and web search tools. The best model, OpenAI o3, answered 46.8% correctly at an average cost of $3.79 per query. Source

Agents must replicate 20 ICML 2024 papers from scratch. Claude 3.5 Sonnet with a basic scaffold scores 21.0%, and o1 with a 36-hour limit scores 26.0%. ML PhDs reached 41.4% on a 3-paper subset after 48 hours. Source

OpenAI open-sources 1,266 hard-to-find, easy-to-verify browsing questions. Deep research answers 51.5%, o1 9.9%, and GPT-4o with browsing 1.9%. Human trainers solved 29.2% within two hours. Source

Agents must exploit real critical-severity vulnerabilities in sandboxed web applications. The best GPT-4o agent framework exploits 12.5% of CVEs with five attempts when given a vulnerability description. Source

METR's paper "Measuring AI Ability to Complete Long Software Tasks" defines the time horizon metric. Claude 3.7 Sonnet had a 50% horizon of around 50 minutes, and the frontier horizon had doubled about every seven months since 2019. Source

CMU researchers release an underspecified variant of SWE-Bench Verified with a simulated user. Models rarely ask questions unless prompted, and most cannot tell a vague issue from a complete one. When they do interact, resolve rates rise sharply. Source

OpenAI publishes 1,488 real freelance software tasks worth $1 million in actual payouts, with a public Diamond split worth $500,800. The best model, Claude 3.5 Sonnet, solves 26.2% of Diamond IC tasks and earns $208,050 across the Diamond set. Source

IBM Research and UIUC publish ITBench, a framework of live Kubernetes scenarios for SRE, compliance (CISO), and FinOps agents. The first paper reports that agents resolve 13.8% of SRE scenarios. Source

MedAgentBench gives agents 300 physician-written tasks in a FHIR-compliant virtual EHR with 100 de-identified patients. Claude 3.5 Sonnet v2 led with a 69.67% success rate; models did much better on record lookups (up to 85%) than on tasks that change records. Source

The maintainers published fixes from a contributor for bugs in the difficulty 4 to 6 cases, so that their programs deploy to AWS without errors, and rewrote these cases with more resources and settings. The dataset card now calls the release v1.1. The results in the NeurIPS 2024 paper use the original release. Source

Microsoft Research and partners publish AIOpsLab, a framework that deploys microservices, injects faults, and evaluates agents on detection, localization, root cause analysis, and mitigation. The best agent in the paper reaches 59.32% accuracy. Source

Paul Gauthier replaces aider's saturating Python-only benchmark with 225 hard Exercism exercises in six languages. o1 with high reasoning effort leads at 61.7%, ahead of Claude 3.5 Sonnet at 45.3%. Source

CMU publishes a simulated software company with GitLab, Plane, ownCloud, RocketChat, and language-model coworkers, plus 175 work tasks. The best agent, OpenHands with Claude 3.5 Sonnet, completes 24.0% of tasks. Source

Seven ML research engineering environments with matched human and agent conditions. Agents score about 4 times the human average at a 2-hour budget, but humans narrowly pass the best agent at 8 hours and reach about twice the agent score at 32 hours. Source

HKU's XLANG Lab and partners release Spider 2.0. The paper reports that its code agent with o1-preview solves 21.3% of tasks, against 91.2% on Spider 1.0 and 73.0% on BIRD. Source

The best setup at launch, o1-preview with the AIDE scaffold, reaches a medal in about 17% of competitions (16.9% in the paper, 17.12% in the repo table). Source

270 tasks from 90 published papers at three difficulty levels. The best agent, CORE-Agent with GPT-4o, reaches 21.48% on the hardest level. Source

Waymo starts publishing its rider-only crash counts, mileage, and human benchmark comparisons with downloadable CSV files, using NHTSA Standing General Order data through June 2024. The hub is updated on the SGO reporting schedule and its release notes record each method change. Source

The best agent (Claude 3.5 Sonnet) solves 17.5% of tasks unguided; GPT-4o solves 12.5%. Agents solve tasks that took human teams up to 11 minutes; the hardest task took humans 24 hours 54 minutes. Source

93 developers screen 1,699 SWE-bench tasks. The 500 tasks that pass become SWE-bench Verified. GPT-4o resolves 33.2% with the best open-source scaffold, double its score on the original SWE-bench. Source

Renmin University and partners release SpreadsheetBench. In the paper GPT-4o scores 18.35% (soft) and 15.02% (hard) overall, and Copilot in Excel about 20% on a subset, while Excel experts score 71.33% and 62.00% on a 50-question subset. Source

ServiceNow Research publishes WorkArena, 33 browser tasks on a live ServiceNow instance, together with the BrowserGym environment for web agents. WorkArena++ (682 compositional tasks) follows in July 2024. Source

GAIA offers 466 questions that need browsing, file reading, and tool use. Humans score 92% while GPT-4 with plugins scores 15%. A leaderboard with private test answers opens on Hugging Face. Source