Benchmarks / PhysicianBench

PhysicianBench

Built by Stanford University · Ruoqi Liu, Imran Q. Mohiuddin, Austin J. Schoeffler, Kavita Renduchintala, et al. · released 4 May 2026

100 physician workflows from Stanford's HealthRex lab, adapted from real e-consults between primary care physicians and specialists across 21 specialties. An agent works in a live FHIR server loaded with a patient's record: it must retrieve data across encounters, reason about the case, place orders, and write the plan. A successor to MedAgentBench from the same group, with longer tasks and physician-validated checkpoints.

Frontier

50%

Pass@1

GPT-5.6-sol · harness: PhysicianBench tool-calling agent loop

13 Aug 2026 · Source: Stanford University (benchmark maintainers)

50.0 ± 2.6 (SD over three runs). Pass@3 62.2, Pass^3 31.0, 43.2 tool calls per task. The site shows no date; the row was added to the site's leaderboard data on 2026-08-13 (repository commit "update leaderboard").

Emerging

The best result is at 20–60% of the ceiling, or at 50–100% of human parity.

See the full leaderboard at Stanford University

Dots are recorded results; the line is the best result so far; the red dot is the current frontier. Sources for every point are in the table below.

Embed this card

The image names the institutions behind the numbers and links back to this page.

HTML:

<a href="https://canagentswork.com/benchmarks/physicianbench/"><img src="https://canagentswork.com/og/benchmarks-physicianbench.png" width="600" height="315" alt="PhysicianBench: the best result is 50% (GPT-5.6-sol, 13 Aug 2026)." loading="lazy"></a>

Markdown:

[![PhysicianBench: the best result is 50% (GPT-5.6-sol, 13 Aug 2026).](https://canagentswork.com/og/benchmarks-physicianbench.png)](https://canagentswork.com/benchmarks/physicianbench/)

What it measures

Pass@1: the share of the 100 tasks fully completed in one attempt, averaged over three independent runs. Each task has 5 to 9 checkpoints (670 in total): FHIR orders are checked against the server state by code, reasoning and documentation by an LLM judge with a clinician-written rubric, and data retrieval by trajectory analysis. A task passes only when every checkpoint passes. The agent uses the benchmark's own tool-calling loop with 14 FHIR search and create tools and at most 100 steps. The leaderboard also reports Pass@3 and Pass^3 (all three runs succeed).

Pass@1: Share of the 100 tasks where every checkpoint passes in a single attempt, mean over three runs. Higher is better.

Status compares the frontier with a ceiling of 100%.

Facts

Grain: Task level: bounded tasks with a clear spec
Environment: api-tools
Grading: state-check, llm-judge, rubric
Tasks: 100
Contamination: Task instructions and checkpoint graders are public in the repository. The patient data is gated. Every task runs in a fresh container, so no state leaks between runs.
Reuse: Code, tasks, and the evaluation harness are Apache 2.0 (repository LICENSE). The patient records ship as a Docker image on Stanford Redivis under a controlled request-and-approval process, governed by the Stanford University Research Data Use Agreement. (cite-only)

Limits to keep in mind

The patient records are neither fully real nor fully synthetic. The paper says they derive from de-identified Stanford STARR records with added perturbations (date shifts, Faker identities, adjusted clinical values). The README says "real patient records"; the site's methodology section says "a realistic synthetic patient record". Both describe the same perturbed real data. Source
Reasoning and documentation checkpoints are graded by an LLM judge against a rubric, so part of the score depends on the judge. Source
One harness only: the maintainers' minimal tool-calling loop. The leaderboard has no dates and no cost figures. All rows are the maintainers' runs. Source
Reliability is low even at the top: the leader's Pass^3 (all three runs succeed) is 31.0% against a Pass@1 of 50.0%. Source

Recorded results

A selection that shows the frontier over time. The full leaderboard is at the source.

System	Pass@1	Date	Source
GPT-5.6-sol · frontier harness: PhysicianBench tool-calling agent loop	50%	13 Aug 2026	Stanford University · primary
Claude Opus 4.6 harness: PhysicianBench tool-calling agent loop	31.7%	4 May 2026	Stanford University · primary
GPT-5.5 harness: PhysicianBench tool-calling agent loop	46.3%	4 May 2026	Stanford University · primary

Timeline

13 Aug 2026 — GPT-5.6-sol reaches 50.0% Pass@1 on PhysicianBench. Source
4 May 2026 — Stanford releases PhysicianBench; GPT-5.5 completes 46.3% of physician e-consult workflows. Source

Where it sits in the atlas

Work ladder: Direct evidence for Healthcare. On the work ladder it counts as 50% of tasks completed. How the ladder works

Work it measures (O*NET work activities): Collect information about patients or clients; Evaluate patient or client condition or treatment options; Order medical tests or procedures; Prescribe medical treatments or devices; Develop patient or client care or treatment plans.

Healthcare

Clinical records work

Go to the source

Website healthrex.github.io
Paper arxiv.org
Full leaderboard healthrex.github.io
Code github.com
Dataset stanford.redivis.com

Last checked 24 Sep 2026 against 5 primary sources. See an error? Tell us.

How to cite

Credit the original work first: PhysicianBench by Stanford University (https://arxiv.org/abs/2605.02240).

Then, if you used this page:

Can Agents Work. "PhysicianBench: frontier results and sources." https://canagentswork.com/benchmarks/physicianbench/ (accessed 2026-09-24). CC BY 4.0.

@misc{caw-physicianbench,
  title        = {{PhysicianBench: frontier results and sources}},
  author       = {{Can Agents Work}},
  year         = {2026},
  howpublished = {\url{https://canagentswork.com/benchmarks/physicianbench/}},
  note         = {Accessed 2026-09-24. CC BY 4.0}
}