Benchmarks / SREGym
SREGym
Built by UIUC and University of Toronto · released 8 May 2026
A live Kubernetes benchmark for SRE agents. Fault injectors break real cloud-native applications (up to 40 microservices), optionally with background noise, and the agent must find the root cause and fix the system. It builds on AIOpsLab and ITBench and adds OS-level, hardware, metastable, and concurrent failures.
Frontier
72.2%
End-to-end success rate
GitHub Copilot (GPT-5.6 Sol, max) · agent: GitHub Copilot
3 Aug 2026 · Source: UIUC (benchmark maintainers)
Full 90-problem suite (cohort SREGym-0508), without noise. Diagnosis 83.3%, mitigation 83.7%. The leaderboard shows no dates; the date is the maintainers' commit that added the row to the site's leaderboard data (SREGym/SREGym-website, lib/leaderboard-data.ts, 2026-08-03). Still the best full-suite entry on 2026-10-10.
The best result is at 60–90% of the ceiling, or at or above human parity.
What it measures
End-to-end success: the agent must both diagnose the root cause correctly (an LLM judge with a fixed checklist) and mitigate the failure (checked programmatically) in the same run. Diagnosis and mitigation rates are also reported, along with time to diagnose, time to mitigate, and tokens. The full suite has 90 problems; SREGym-Lite is a 21-problem subset.
End-to-end success rate: Share of runs where diagnosis and mitigation both succeed on the 90-problem suite, without injected noise. Higher is better.
Status compares the frontier with a ceiling of 100%.
Facts
- Grain
- Task level: bounded tasks with a clear spec
- Environment
- live-system, cli
- Grading
- state-check, llm-judge
- Tasks
- 90
- Reuse
- MIT (GitHub repository license). Paper is CC BY 4.0. (open-mit)
Limits to keep in mind
- Diagnosis is graded by an LLM judge (Claude Sonnet 4.6 in the paper), which adds variance that programmatic checks do not have. Source
- Deployed applications are far smaller than production systems; the largest has 40 microservices. Source
- Scores fall with injected noise. Claude Code with Sonnet 4.6 drops from 60.7% to 53.7% end to end. Leaderboard entries for newer models are without noise. Source
- SREGym-Lite leaderboard scores (best 96.1%) use a 17-problem subset and are not comparable with the 90-problem suite. The maintainers replaced the earlier 21-problem Lite cohort in October 2026. Source
Recorded results
A selection that shows the frontier over time. The full leaderboard is at the source.
Timeline
- 8 May 2026 — SREGym paper released with 90 live SRE problems. Source
Where it sits in the atlas
Work ladder: Direct evidence for DevOps, SRE, and IT operations. On the work ladder it counts as 72.2% of tasks completed. How the ladder works
Work it measures (O*NET work activities): Monitor operation of computer or information technologies; Resolve computer problems.
Go to the source
- Website sregym.com
- Paper arxiv.org
- Full leaderboard sregym.com
- Code github.com
Last checked 10 Oct 2026 against 8 primary sources. See an error? Tell us.
How to cite
Credit the original work first: SREGym by UIUC and University of Toronto (https://arxiv.org/abs/2605.07161).
Then, if you used this page:
Can Agents Work. "SREGym: frontier results and sources." https://canagentswork.com/benchmarks/sregym/ (accessed 2026-10-11). CC BY 4.0.@misc{caw-sregym,
title = {{SREGym: frontier results and sources}},
author = {{Can Agents Work}},
year = {2026},
howpublished = {\url{https://canagentswork.com/benchmarks/sregym/}},
note = {Accessed 2026-10-11. CC BY 4.0}
}