CVE-Bench

Built by UIUC · released 31 Mar 2025

40 critical-severity CVEs (CVSS 9.0 or higher) in real web applications such as WordPress plugins and LLM tools. Each runs in an isolated sandbox. An agent must carry out a real attack, for example read the secret table, log in as admin, or execute code, and an evaluator checks the effect.

Frontier

12.5%

Success rate (one-day, 5 attempts)

AutoGPT with GPT-4o · agent: AutoGPT

Mar 2025 · Source: UIUC (benchmark maintainers)

Best result in the paper: 12.5% one-day and 10% zero-day with five attempts (abstract rounds to 13%). Measured on CVE-Bench v1 before the v2.0 grading fixes.

Open

The best result is below 20% of the ceiling, or below half of human parity.

Dots are recorded results; the line is the best result so far; the red dot is the current frontier. Sources for every point are in the table below.

Embed this card

The image names the institutions behind the numbers and links back to this page.

HTML:

<a href="https://canagentswork.com/benchmarks/cve-bench/"><img src="https://canagentswork.com/og/benchmarks-cve-bench.png" width="600" height="315" alt="CVE-Bench: the best result is 12.5% (AutoGPT with GPT-4o, Mar 2025)." loading="lazy"></a>

Markdown:

[![CVE-Bench: the best result is 12.5% (AutoGPT with GPT-4o, Mar 2025).](https://canagentswork.com/og/benchmarks-cve-bench.png)](https://canagentswork.com/benchmarks/cve-bench/)

What it measures

Whether an agent can exploit a real web vulnerability end to end. In the zero-day setting the agent gets only the target; in the one-day setting it also gets a short description of the vulnerability. Success is one of eight attack outcomes checked by the sandbox. The paper reports success with one attempt and with five attempts. The headline here is one-day success with five attempts.

Success rate (one-day, 5 attempts): Share of the 40 CVEs exploited within five attempts when the agent is given a vulnerability description. Higher is better.

Status compares the frontier with a ceiling of 100%.

Facts

Grain: Task level: bounded tasks with a clear spec
Environment: cli, browser
Grading: state-check
Tasks: 40
Contamination: CVEs are public in the National Vulnerability Database with published details. Since v2.0, secrets are generated at runtime instead of hard-coded, in case old secrets appear in training data.
Reuse: Apache-2.0 (GitHub repository license). Reference exploits are available on request. (open-apache)

Limits to keep in mind

Published results use gpt-4o-2024-11-20 with three agent frameworks from early 2025. There is no public leaderboard for newer models. Source
v2.0 (2025-10-30) closed two shortcuts that agents used to pass tasks. The maintainers say GPT-4o agent success rates fell by up to 32.5 points after the fixes, so paper numbers overstate v1 ability. Source
v2.1.0 (2026-01-12) replaced arbitrary file upload with remote code execution as an attack goal, so results across versions are not directly comparable. Source
40 tasks is a small set; one task equals 2.5 percentage points. Source

Recorded results

A selection that shows the frontier over time. The full leaderboard is at the source.

System	Success rate (one-day, 5 attempts)	Date	Source
Cybench agent (Cy-Agent) with GPT-4o	2.5%	Mar 2025	UIUC · primary
AutoGPT with GPT-4o · frontier	12.5%	Mar 2025	UIUC · primary

Timeline

30 Oct 2025 — CVE-Bench v2.0 closes grading shortcuts. Source
31 Mar 2025 — UIUC releases CVE-Bench with 40 critical web CVEs. Source

Where it sits in the atlas

Security

Vulnerability research

Go to the source

Paper arxiv.org
Code github.com
Announcement open.substack.com

Last checked 23 Sep 2026 against 4 primary sources. See an error? Tell us.

How to cite

Credit the original work first: CVE-Bench by UIUC (https://arxiv.org/abs/2503.17332).

Then, if you used this page:

Can Agents Work. "CVE-Bench: frontier results and sources." https://canagentswork.com/benchmarks/cve-bench/ (accessed 2026-09-24). CC BY 4.0.

@misc{caw-cve-bench,
  title        = {{CVE-Bench: frontier results and sources}},
  author       = {{Can Agents Work}},
  year         = {2026},
  howpublished = {\url{https://canagentswork.com/benchmarks/cve-bench/}},
  note         = {Accessed 2026-09-24. CC BY 4.0}
}