Benchmarks / MedAgentBench
MedAgentBench
Built by Stanford University · Yixing Jiang, Kameron C. Black, Gloria Geng, Danny Park, et al. · released 24 Jan 2025
A Stanford benchmark of 300 clinical tasks that a physician might hand to an assistant inside an electronic health record. An agent works through a FHIR API against a virtual EHR that holds 100 de-identified patient profiles, for example to look up a lab value, record vitals, or place an order.
Frontier
69.7%
Task success rate
Claude 3.5 Sonnet v2 · harness: MedAgentBench baseline orchestrator
24 Jan 2025 · Source: Stanford University (benchmark maintainers)
Table 3 of the paper. Query tasks 85.33, action tasks 54.00. Temperature 0, single attempt. No newer primary results exist.
The best result is at 60–90% of the ceiling, or at or above human parity.
What it measures
Task success on one try (pass@1). Two internal medicine physicians wrote the 300 tasks in 10 categories, from patient lookup and lab retrieval to ordering tests, referrals, and medications. Half the tasks only need GET requests (query tasks); half need POST requests that change records (action tasks). Query tasks are graded against reference answers, and action tasks by rule-based checks of the request payload. The agent gets nine FHIR functions and at most eight rounds.
Task success rate: Share of the 300 tasks that the agent completes correctly on a single attempt. Higher is better.
Status compares the frontier with a ceiling of 100%.
Facts
- Grain
- Task level: bounded tasks with a clear spec
- Environment
- api-tools
- Grading
- automated-tests, state-check
- Tasks
- 300
- Contamination
- Tasks and the FHIR environment are public. Timestamps in the patient data are jittered at the patient level.
- Reuse
- Code is MIT (repository license). The arXiv paper is CC BY 4.0. Patient data derives from de-identified Stanford STARR records and ships as a Docker image; the reference solutions are distributed separately from a Stanford Medicine Box link. (cite-only)
Limits to keep in mind
- Published scores are from the January 2025 paper (Claude 3.5 Sonnet v2, GPT-4o, DeepSeek-V3 and others). There is no maintained leaderboard, so newer models are not tracked here. Source
- The baseline agent is a simple orchestrator with nine FHIR functions and an eight-round cap. Many failures were format errors, for example answering in a sentence instead of a bare value. Source
- Tasks take only a few steps each. The authors kept the first release to 300 tasks and 100 patients to limit evaluation cost. Source
Recorded results
A selection that shows the frontier over time. The full leaderboard is at the source.
| System | Task success rate | Date | Source |
|---|---|---|---|
| DeepSeek-V3 harness: MedAgentBench baseline orchestrator | 62.7% | 24 Jan 2025 | Stanford University · primary |
| GPT-4o harness: MedAgentBench baseline orchestrator | 64% | 24 Jan 2025 | Stanford University · primary |
| Claude 3.5 Sonnet v2 · frontier harness: MedAgentBench baseline orchestrator | 69.7% | 24 Jan 2025 | Stanford University · primary |
Timeline
- 24 Jan 2025 — Stanford releases MedAgentBench. Source
Where it sits in the atlas
Go to the source
Last checked 23 Sep 2026 against 4 primary sources. See an error? Tell us.
How to cite
Credit the original work first: MedAgentBench by Stanford University (https://arxiv.org/abs/2501.14654).
Then, if you used this page:
Can Agents Work. "MedAgentBench: frontier results and sources." https://canagentswork.com/benchmarks/medagentbench/ (accessed 2026-09-24). CC BY 4.0.@misc{caw-medagentbench,
title = {{MedAgentBench: frontier results and sources}},
author = {{Can Agents Work}},
year = {2026},
howpublished = {\url{https://canagentswork.com/benchmarks/medagentbench/}},
note = {Accessed 2026-09-24. CC BY 4.0}
}