Benchmarks / Vulnerability research

Vulnerability research

Find, reproduce, or exploit vulnerabilities, for example CTF challenges or CVEs.

Part of Security.

• ARTEMIS live penetration test (agents vs. professionals)by Stanford University, Carnegie Mellon University and Gray Swan AIProject level · Total score (complexity plus weighted severity of valid findings)

  95.2ARTEMIS A2 (ensemble supervisor, Claude Sonnet 4 sub-agents) · 10 Dec 2025

  Strong
• CVE-Benchby UIUCTask level · Pass@1 (one-day)

  40%Default Agent + Claude Opus 4.6 · 11 Apr 2026

  Emerging
• Cybenchby Stanford UniversityTask level · Unguided % solved

  100%Claude Mythos Preview · 7 Apr 2026

  Saturated
• CyberBench v1.1by Vals AITask level · Overall accuracy (mean of PoC and Patch pass rates)

  78%GPT-6 Sol · 7 Oct 2026

  Strong
• CyberGymby UC BerkeleyTask level · Success rate (Level 1)

  99.2%Lyrie Agent (multi-model) · 15 Sep 2026

  Saturated
• DARPA AI Cyber Challenge, Final Competitionby DARPA and ARPA-HProject level · Synthetic vulnerabilities found (scored round)

  43Team Atlanta · 8 Aug 2025

  Strong