Benchmarks / AutomationBench
AutomationBench
Built by Zapier · Daniel Shepard, Robin Salimans · released 21 Apr 2026
Zapier's test of whether an agent can carry out a cross-app business workflow on its own. Each task boots a small simulated company (CRM, inbox, calendar, sheets, helpdesk) in 47 simulated SaaS apps, hands the agent one trigger message, and checks the state it leaves behind. Tasks come from workflow patterns on Zapier's platform in sales, marketing, operations, support, finance, and HR.
Frontier
51.3%
Pass rate (task_completed_correctly)
Gemini 4 Argon (High)
30 Sep 2026 · Source: Zapier (benchmark maintainers)
First on board version 1.0.6 at $1.70 per task at standard list pricing (Zapier marks the Gemini costs as promotional; $0.85 at the promotional price). No fallback model. Gemini 4 Argon (Medium) is second at 50.08%. The board shows no date; we use the date of Google's launch post (2026-09-30), which reports the same entry as #1 at 51.3%. At launch the model was available only to selected cyber defenders in Google's Fairwind Program, not to the public. Run-to-run variance is typically within 1% (single run). Mistral's launch post for Mistral Large 4 (public preview, 2026-10-06) reports 59.9% on the 657 workflows; it is lab-reported, the post does not say who ran it, and the model is not on Zapier's board as of 2026-10-10, so it is not the frontier.
The best result is at 20–60% of the ceiling, or at 50–100% of human parity.
What it measures
Strict pass rate: a task counts only when every deterministic assertion on the final state holds, including negative assertions (for example no email to the wrong team). The agent finds endpoints with a search tool and calls them with an execute tool, up to 50 steps, with no human in the loop. Official scores come from a held-out private set of about 657 tasks that mirrors the public set but is harder. A partial-credit score (share of assertions passed) is reported for diagnosis only.
Pass rate (task_completed_correctly): Share of private-set tasks where every scored assertion on the final environment state passes. Higher is better.
Status compares the frontier with a ceiling of 100%.
Facts
- Grain
- Task level: bounded tasks with a clear spec
- Environment
- api-tools, simulated-workplace
- Grading
- state-check
- Tasks
- 657
- Contamination
- Leaderboard scores use a held-out private task set. The public set is for research and its scores differ from the board by design (for example Claude Opus 5 at 50.3% on the public set).
- Reuse
- The 600 public tasks (100 per domain) and the harness ship in the repository under the MIT license. The private leaderboard set is not released. (open-mit)
Limits to keep in mind
- Several high entries (Claude Opus 5.5 and Sonnet 5.5) are "default fallbacks" runs: tasks that a model refused were rerun through the provider's default fallback model, and the rerun result counts. Zapier ranks these entries, but a no-fallback run scores lower (Anthropic reports 40.0% for Opus 5.5 without fallbacks, against 42.47% with them). The current leader, Gemini 4 Argon, ran without a fallback. Source
- Version updates change the tasks. In 1.0.6 (2026-07-31) Zapier fixed task fairness, made private tasks harder, and reran the official entries. Scores before and after a version change are not directly comparable, and the launch paper's scores (best 9.9%) predate 1.0.5. Source
- Tasks and environments are synthetic (generated with models, then hardened and sampled for manual review), so some tasks may be unrealistic or impossible. Zapier says bugs remain. Source
- Scores are from a single run per model. Zapier says run-to-run variance is typically within 1%. Source
Recorded results
A selection that shows the frontier over time. The full leaderboard is at the source.
Timeline
- 30 Sep 2026 — Gemini 4 Argon tops AutomationBench 1.0.6 at 51.29%, the first score above 50%. Source
- Sep 2026 — Claude Opus 5.5 tops AutomationBench 1.0.6 at 42.47%, up from 9.9% at launch. Source
- 31 Jul 2026 — AutomationBench 1.0.6 fixes task fairness, hardens private tasks, and reruns official entries. Source
- 21 Apr 2026 — Zapier publishes AutomationBench; best model completes 9.9% of cross-app workflows. Source
Where it sits in the atlas
Work ladder: Direct evidence for Management and business operations. On the work ladder it counts as 51.3% of tasks completed. How the ladder works
Work it measures (O*NET work activities): Implement procedures or processes; Maintain operational records; Maintain sales or financial records; Communicate with others about operational plans or activities; Respond to customer problems or inquiries; Perform administrative or clerical activities.
Management and business operationsSales and marketing (partial)Customer support (partial)Finance and accounting (partial)Office and administrative support (partial)
Business workflow automationEnterprise workflowsCRM operationsCustomer service
Go to the source
- Website zapier.com
- Paper arxiv.org
- Full leaderboard zapier.com
- Code github.com
Last checked 10 Oct 2026 against 8 primary sources. See an error? Tell us.
How to cite
Credit the original work first: AutomationBench by Zapier (https://arxiv.org/abs/2604.18934).
Then, if you used this page:
Can Agents Work. "AutomationBench: frontier results and sources." https://canagentswork.com/benchmarks/automationbench/ (accessed 2026-10-11). CC BY 4.0.@misc{caw-automationbench,
title = {{AutomationBench: frontier results and sources}},
author = {{Can Agents Work}},
year = {2026},
howpublished = {\url{https://canagentswork.com/benchmarks/automationbench/}},
note = {Accessed 2026-10-11. CC BY 4.0}
}