Benchmarks / MedAgentBench

MedAgentBench

Built by Stanford University · Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, et al. · released 24 Jan 2025

A Stanford benchmark of 300 clinical tasks that a physician might hand to an assistant inside an electronic health record. An agent works through a FHIR API against a virtual EHR that holds 100 de-identified patient profiles, for example to look up a lab value, record vitals, or place an order.

Frontier

69.7%

Task success rate

Claude 3.5 Sonnet v2 · harness: MedAgentBench baseline orchestrator

24 Jan 2025 · Source: Stanford University (benchmark maintainers)

Table 3 of the paper (the same in the NEJM AI version). Query tasks 85.33, action tasks 54.00. Temperature 0, single attempt. No newer result under the original protocol exists; the 91.0% of the MedAgentBench v2 agent design (GPT-4.1, PSB 2026) uses added tools and task-specific prompts.

Strong

The best result is at 60–90% of the ceiling, or at or above human parity.

Dots are recorded results; the line is the best result so far; the red dot is the current frontier. Sources for every point are in the table below.

Embed this card

The image names the institutions behind the numbers and links back to this page.

HTML:

<a href="https://canagentswork.com/benchmarks/medagentbench/"><img src="https://canagentswork.com/og/benchmarks-medagentbench.png" width="600" height="315" alt="MedAgentBench: the best result is 69.7% (Claude 3.5 Sonnet v2, 24 Jan 2025)." loading="lazy"></a>

Markdown:

[![MedAgentBench: the best result is 69.7% (Claude 3.5 Sonnet v2, 24 Jan 2025).](https://canagentswork.com/og/benchmarks-medagentbench.png)](https://canagentswork.com/benchmarks/medagentbench/)

What it measures

Task success on one try (pass@1). Two internal medicine physicians wrote the 300 tasks in 10 categories, from patient lookup and lab retrieval to ordering tests, referrals, and medications. Half the tasks only need GET requests (query tasks); half need POST requests that change records (action tasks). Query tasks are graded against reference answers, and action tasks by rule-based checks of the request payload. The agent gets nine FHIR functions and at most eight rounds.

Task success rate: Share of the 300 tasks that the agent completes correctly on a single attempt. Higher is better.

Status compares the frontier with a ceiling of 100%.

Facts

Grain: Task level: bounded tasks with a clear spec
Environment: api-tools
Grading: automated-tests
Tasks: 300
Contamination: Tasks and the FHIR environment are public. Timestamps in the patient data are jittered at the patient level.
Reuse: Code is MIT (repository license). The arXiv paper is CC BY 4.0. Patient data derives from de-identified Stanford STARR records and ships as a Docker image; the reference solutions are distributed separately from a Stanford Medicine Box link. (cite-only)

Limits to keep in mind

Published scores are from the January 2025 paper (Claude 3.5 Sonnet v2, GPT-4o, DeepSeek-V3 and others); the NEJM AI version of August 2025 reports the same table. There is no maintained leaderboard, so newer models are not tracked here. Source
A later paper with the Stanford authors (MedAgentBench v2, PSB 2026) reached 91.0% with GPT-4.1 by adding structured FHIR tools, a calculator, a finish tool, and few-shot prompts. That changes the tool set that the original protocol fixes, so we do not show it as the frontier. Source
The baseline agent is a simple orchestrator with nine FHIR functions and an eight-round cap. Many failures were format errors, for example answering in a sentence instead of a bare value. Source
Grader problems: a third-party audit reports that the graders of tasks 5 and 9 enforce answer formats that the task context does not state, so clinically correct answers fail. Tasks are short (2.3 steps on average in the NEJM AI paper). Source

Recorded results

A selection that shows the frontier over time. The full leaderboard is at the source.

System	Task success rate	Date	Source
DeepSeek-V3 harness: MedAgentBench baseline orchestrator	62.7%	24 Jan 2025	Stanford University · primary
GPT-4o harness: MedAgentBench baseline orchestrator	64%	24 Jan 2025	Stanford University · primary
Claude 3.5 Sonnet v2 · frontier harness: MedAgentBench baseline orchestrator	69.7%	24 Jan 2025	Stanford University · primary

Timeline

4 May 2026 — A successor to MedAgentBench, PhysicianBench, launches with 100 physician e-consult workflows. Source
24 Jan 2025 — Stanford releases MedAgentBench. Source

Where it sits in the atlas

Work ladder: Direct evidence for Healthcare. On the work ladder it counts as 69.7% of tasks completed. How the ladder works

Work it measures (O*NET work activities): Maintain health or medical records; Collect information about patients or clients; Prescribe medical treatments or devices.

Healthcare

Clinical records work

Go to the source

Website stanfordmlgroup.github.io
Paper arxiv.org
Code github.com

Last checked 10 Oct 2026 against 7 primary sources. See an error? Tell us.

How to cite

Credit the original work first: MedAgentBench by Stanford University (https://arxiv.org/abs/2501.14654).

Then, if you used this page:

Can Agents Work. "MedAgentBench: frontier results and sources." https://canagentswork.com/benchmarks/medagentbench/ (accessed 2026-10-11). CC BY 4.0.

@misc{caw-medagentbench,
  title        = {{MedAgentBench: frontier results and sources}},
  author       = {{Can Agents Work}},
  year         = {2026},
  howpublished = {\url{https://canagentswork.com/benchmarks/medagentbench/}},
  note         = {Accessed 2026-10-11. CC BY 4.0}
}