Can Agents Work?

Benchmarks / Code Review Bench

Code Review Bench (CRB)

Built by Martian · Aleksandr Zverianskii, Ashley Zhang, Jacob Clyne, Antía Garcia, et al. · released 26 Feb 2026

Open benchmark for AI code review tools. The offline set runs each tool on the same 50 real pull requests and scores its comments against human-curated golden comments with an LLM judge. A separate online set samples fresh GitHub pull requests where review bots commented and checks which bot suggestions the developer then fixed.

Frontier

65.1

F2 score (Core profile, offline set)

Qodo Extended

20 Aug 2026 · Source: Martian (benchmark maintainers)

Offline set, Core profile, judge Claude Opus 4.5 (dashboard default). Precision 67.1, recall 64.6, F1 65.8 across 50 PRs. Under the Claude Sonnet 4.5 judge Qodo Extended scores 56.8 F2 and under GPT-5.2 59.3 F2, where Cubic v2 leads (59.9 under both).

Strong

The best result is at 60–90% of the ceiling, or at or above human parity.

See the full leaderboard at Martian

Code Review Bench: F2 score (Core profile, offline set) over time, 5 recorded results. 020406080100Aug 2026Aug 2026Aug 2026Aug 2026Aug 2026Sep 2026 Qodo Extended: 65.1 (20 Aug 2026) Cubic v2: 64.9 (20 Aug 2026) Augment: 64 (20 Aug 2026) GitHub Copilot: 58 (20 Aug 2026) CodeRabbit: 51.1 (20 Aug 2026)
Dots are recorded results; the line is the best result so far; the red dot is the current frontier. Sources for every point are in the table below.
Embed this card

The image names the institutions behind the numbers and links back to this page.

HTML:

<a href="https://canagentswork.com/benchmarks/martian-code-review-bench/"><img src="https://canagentswork.com/og/benchmarks-martian-code-review-bench.png" width="600" height="315" alt="Code Review Bench: the best result is 65.1 (Qodo Extended, 20 Aug 2026)." loading="lazy"></a>

Markdown:

[![Code Review Bench: the best result is 65.1 (Qodo Extended, 20 Aug 2026).](https://canagentswork.com/og/benchmarks-martian-code-review-bench.png)](https://canagentswork.com/benchmarks/martian-code-review-bench/)

What it measures

Whether a code review tool finds the real issues in a pull request without adding noise. Offline: 50 PRs from Sentry, Grafana, Cal.com, Discourse, and Keycloak, with 173 golden comments labelled by severity and category. Precision is the share of tool comments that match a golden comment. Recall is the share of golden comments (in the active category profile) that the tool found. The dashboard default combines them as F2 (recall weighted 4x) on the Core profile (158 golden comments: bug, security, concurrency, data, API, perf, test gap, doc defect). Online: per-PR precision and recall against the developer's post-review fixes.

F2 score (Core profile, offline set): F-beta with beta 2 on a 0 to 100 scale, computed from precision and recall against golden comments on the Core category profile. Judge model Claude Opus 4.5 (the dashboard default). Higher is better.

Status compares the frontier with a ceiling of 100.

Facts

Grain
Task level: bounded tasks with a clear spec
Environment
repo
Grading
llm-judge
Tasks
50
Human reference
Golden comments come from human reviewers, so the ceiling is the human-curated issue set. No human reviewer is scored as a row.
Contamination
The offline PRs come from well-known public repositories, so tools may have seen them in training. Martian runs the online set on fresh PRs to check for this.
Reuse
MIT (repo license covers PRs metadata, golden comments, judge prompts, pipeline, and results) (open-mit)

Limits to keep in mind

  • Static offline dataset of 50 PRs from public repos. Maintainers say tools may have seen these PRs during training and point to the online set as the check. Source
  • Scores depend on the LLM judge. Under the Claude Opus 4.5 judge, Qodo Extended leads; under Claude Sonnet 4.5 and GPT-5.2, Cubic v2 leads. Maintainers say the top 5 tools are the same across judges and most tools move at most 2 ranks. Source
  • The golden set has sparse coverage of style and nit issues (about 10 across 50 PRs), so correct style comments can still count as false positives. Source
  • Rows are review products and coding assistants (for example Qodo, CodeRabbit, Claude Code), not raw models. Martian runs every tool itself and only publishes tools with enough public GitHub usage and attributable bot accounts. Source

Recorded results

A selection that shows the frontier over time. The full leaderboard is at the source.

SystemF2 score (Core profile, offline set)DateSource
Qodo Extended · frontier65.120 Aug 2026Martian · primary
Cubic v264.920 Aug 2026Martian · primary
Augment6420 Aug 2026Martian · primary
GitHub Copilot5820 Aug 2026Martian · primary
CodeRabbit51.120 Aug 2026Martian · primary

Timeline

  • 26 Feb 2026 — Martian launches Code Review Bench. Source

Where it sits in the atlas

Software engineering

Code review

Last checked 23 Sep 2026 against 5 primary sources, with a second independent check. See an error? Tell us.

How to cite

Credit the original work first: Code Review Bench by Martian.

Then, if you used this page:

Can Agents Work. "Code Review Bench: frontier results and sources." https://canagentswork.com/benchmarks/martian-code-review-bench/ (accessed 2026-09-24). CC BY 4.0.
@misc{caw-martian-code-review-bench,
  title        = {{Code Review Bench: frontier results and sources}},
  author       = {{Can Agents Work}},
  year         = {2026},
  howpublished = {\url{https://canagentswork.com/benchmarks/martian-code-review-bench/}},
  note         = {Accessed 2026-09-24. CC BY 4.0}
}