Benchmarks / CyberGym
CyberGym
Built by UC Berkeley · released 3 Jun 2025
1,507 real vulnerabilities from 188 open-source projects, drawn from Google's OSS-Fuzz. Given a text description of a bug and the unpatched codebase, an agent must write a proof-of-concept input that triggers the vulnerability. UC Berkeley maintains a public leaderboard with team submissions.
Frontier
99.2%
Success rate (Level 1)
Lyrie Agent (multi-model) · agent: Lyrie Agent
15 Sep 2026 · Source: Lyrie (reported by the system's developer)
Submitted by Lyrie and listed first on the official leaderboard with the "dynamic" label (runs against a cleaned vulnerable Docker image), one trial. Teams evaluate and submit their own results; the submission rules ask for sample trajectories from at least 10 tasks for the maintainers to review and require the final-submission metric. Lyrie's writeup reports 1,495 of 1,507 tasks with one designated PoC per task; 894 of the solves came from libFuzzer runs on model-written seed inputs and 601 from a model-driven generator loop. The previous leader, Creation (天工) from Creation Lab, scored 98.47% (dated 2026-09-07 on the board). Lyrie's writeup page is dated 2026-09-24.
The best result is at 90% or more of the ceiling.
What it measures
Whether an agent can reproduce a known vulnerability in a large real codebase. A task counts as solved when the generated proof of concept crashes the pre-patch build but not the post-patch build. The headline (Level 1) gives the agent the vulnerability description and the source. Other levels give less or more information. The leaderboard notes when a team used a runnable vulnerable image or test-time memory.
Success rate (Level 1): Share of the 1,507 instances where the agent produces a working proof of concept for the target vulnerability. Higher is better.
Status compares the frontier with a ceiling of 100%.
Facts
- Grain
- Task level: bounded tasks with a clear spec
- Environment
- repo, cli
- Grading
- automated-tests
- Tasks
- 1,507
- Contamination
- Vulnerabilities and their fixes are public in OSS-Fuzz and project histories, so training data may include them.
- Reuse
- Not stated. The Hugging Face dataset card (about 240 GB) gives no license, and the tasks come from OSS-Fuzz projects under their own licenses. (cite-only)
Limits to keep in mind
- Leaderboard results are run and submitted by the teams themselves; the maintainers ask for sample trajectories from at least 10 tasks to review. They also say the most capable systems "have nearly saturated the benchmark". Source
- Agent runs are stochastic and vulnerability descriptions can be ambiguous. The maintainers say small score differences may not reflect real capability gaps now that leading systems score high. Source
- Top entries use multiple models, orchestration, runnable vulnerable images, and test-time memory across instances. They measure agent systems, not single models. Source
- The task is reproduction of known bugs, not discovery. The paper separately reports 34 zero-days found by agents in open-ended runs. Source
Recorded results
A selection that shows the frontier over time. The full leaderboard is at the source.
| System | Success rate (Level 1) | Date | Source |
|---|---|---|---|
| Lyrie Agent (multi-model) · frontier | 99.2% | 15 Sep 2026 | Lyrie · lab-reported |
| MDASH (multi-model) | 91% | 17 Jun 2026 | Microsoft Research · lab-reported |
| Claude Mythos Preview | 83.1% | 7 Apr 2026 | Anthropic · lab-reported |
| OpenHands (GPT-5) | 39.4% | 5 Dec 2025 | UC Berkeley · primary |
| OpenHands (Claude Sonnet 3.7) | 11.9% | 15 May 2025 | UC Berkeley · primary |
Timeline
- 7 Sep 2026 — CyberGym leader reaches 98.5% reproduction rate. Source
- 4 Aug 2026 — CyberGym maintainers say top systems have nearly saturated the benchmark. Source
- 3 Jun 2026 — The CyberGym team releases CyberGym-E2E, which adds bug discovery and patching to proof-of-concept work. Source
- 3 Jun 2025 — UC Berkeley releases CyberGym with 1,507 real vulnerabilities. Source
Where it sits in the atlas
Work ladder: Direct evidence for Security. On the work ladder it counts as 99.2% of tasks completed. How the ladder works
Work it measures (O*NET work activities): Test performance of computer or information systems.
Go to the source
- Website www.cybergym.io
- Paper arxiv.org
- Full leaderboard www.cybergym.io
- Code github.com
- Dataset huggingface.co
Last checked 10 Oct 2026 against 11 primary sources. See an error? Tell us.
How to cite
Credit the original work first: CyberGym by UC Berkeley (https://arxiv.org/abs/2506.02548).
Then, if you used this page:
Can Agents Work. "CyberGym: frontier results and sources." https://canagentswork.com/benchmarks/cybergym/ (accessed 2026-10-11). CC BY 4.0.@misc{caw-cybergym,
title = {{CyberGym: frontier results and sources}},
author = {{Can Agents Work}},
year = {2026},
howpublished = {\url{https://canagentswork.com/benchmarks/cybergym/}},
note = {Accessed 2026-10-11. CC BY 4.0}
}