Benchmarks / Long-horizon autonomy

Long-horizon autonomy

How long a task (in human time) an agent can complete on its own.

• AISI multi-step cyber attack rangesby AI Security InstituteProject level · Average steps completed, "The Last Ones" (100M tokens)

  22Claude Mythos Preview (early checkpoint) · 13 Apr 2026

  Strong
• ARTEMIS live penetration test (agents vs. professionals)by Stanford University, Carnegie Mellon University and Gray Swan AIProject level · Total score (complexity plus weighted severity of valid findings)

  95.2ARTEMIS A2 (ensemble supervisor, Claude Sonnet 4 sub-agents) · 10 Dec 2025

  Strong
• BEHAVIOR Challenge 2025by Stanford UniversityTask level · Q-score (held-out test)

  0.26Robot Learning Collective · 14 Nov 2025

  Emerging
• CodeClashby Stanford University and Princeton UniversityProject level · Elo (all arenas)

  1385 EloClaude Sonnet 4.5 · 3 Nov 2025

  Unrated
• METR Task-Completion Time Horizonsby METRTask level · 50% task-completion time horizon

  17 hClaude Mythos Preview (early) · 7 Apr 2026

  Unrated
• RE-Benchby METRTask level · Human-expert percentile matched (8-hour budget)

  37%Claude 3.5 Sonnet (New) (Modular scaffold) · 31 Jan 2025

  Emerging
• TheAgentCompanyby Carnegie Mellon UniversityTask level · Tasks resolved

  42.9%TTE-MatrixAgent + DeepSeek-V3.2 · 10 Nov 2025

  Emerging
• Vending-Bench 2by Andon LabsProject level · Final bank balance

  $15,515GPT-6 Astra · 7 Sep 2026

  Unrated