Benchmarks / Ambig-SWE
Ambig-SWE
Built by Carnegie Mellon University · Sanidhya Vijayvargiya, Xuhui Zhou, Akhila Yerukola, Maarten Sap, et al. · released 18 Feb 2025
An underspecified variant of SWE-Bench Verified. Each of the 500 GitHub issues is rewritten by GPT-4o to drop key details. The agent, running in OpenHands, can ask a simulated user (GPT-4o with the full issue) for clarification. The study asks whether agents notice missing information, ask useful questions, and use the answers to fix the issue.
Frontier
89%
Underspecificity detection accuracy (best prompt)
Claude Sonnet 4 · harness: OpenHands
21 Feb 2026 · Source: Carnegie Mellon University (benchmark maintainers)
Table 2, Strong Encouragement prompt: accuracy 0.89, FPR 0.03, FNR 0.18. Resolve rate on underspecified issues 61.40% with interaction versus 40.00% without and 68.00% with the full issue (Figure 3). Date is the v3 paper date (ICLR 2026 version).
The best result is at 60–90% of the ceiling, or at or above human parity.
What it measures
Three steps. (1) Detection: given a mix of full and underspecified issues, does the agent ask for clarification only when it needs to? Scored as accuracy with false positive and false negative rates, under three prompts of rising encouragement to interact. (2) Question quality. (3) Resolve rate on underspecified issues in three settings: Hidden (no interaction), Interaction (agent may ask the user proxy), and Full (original issue). The headline here is the detection accuracy under the model's best prompt, because it measures the clarification behavior directly. Chance is 50%.
Underspecificity detection accuracy (best prompt): Accuracy at deciding whether an issue needs clarification, over a balanced mix of full and underspecified SWE-Bench Verified issues. Table 2 of the paper. Chance is 50%. Higher is better.
Status compares the frontier with a ceiling of 100%.
Facts
- Grain
- Task level: bounded tasks with a clear spec
- Environment
- repo, chat
- Grading
- automated-tests, outcome-metric
- Tasks
- 500
- Contamination
- Built on SWE-Bench Verified, which models may have trained on. The authors note that some models do well in the Hidden setting possibly through data leakage.
- Reuse
- Paper CC BY 4.0; repo MIT. Issues derive from SWE-Bench Verified. (open-mit)
Limits to keep in mind
- Underspecified issues are synthetic GPT-4o summaries of well-specified issues, and the user is a GPT-4o proxy, not a real developer. Source
- Detection results depend heavily on the prompt. Claude Sonnet 4 scores 74% with a neutral prompt and 89% with strong encouragement; Claude Sonnet 3.5 peaks at 84% with moderate encouragement and drops to 76% with strong encouragement. Source
- Six models were tested (Claude Sonnet 4, Claude Sonnet 3.5, Claude Haiku 3.5, Qwen 3 Coder 480B, Deepseek-v2, Llama 3.1 70B). No models newer than mid-2025 appear in the ICLR 2026 version. Source
- Claude Sonnet 4 was run on 100 of 500 instances in the Hidden setting because of cost. Source
Recorded results
A selection that shows the frontier over time. The full leaderboard is at the source.
| System | Underspecificity detection accuracy (best prompt) | Date | Source |
|---|---|---|---|
| Claude Sonnet 3.5 harness: OpenHands | 84% | 21 Feb 2026 | Carnegie Mellon University · primary |
| Claude Sonnet 4 · frontier harness: OpenHands | 89% | 21 Feb 2026 | Carnegie Mellon University · primary |
| Qwen 3 Coder 480B harness: OpenHands | 50% | 21 Feb 2026 | Carnegie Mellon University · primary |
Timeline
- 18 Feb 2025 — Ambig-SWE tests whether coding agents ask for clarification. Source
Where it sits in the atlas
Go to the source
Last checked 23 Sep 2026 against 3 primary sources. See an error? Tell us.
How to cite
Credit the original work first: Ambig-SWE by Carnegie Mellon University (https://arxiv.org/abs/2502.13069).
Then, if you used this page:
Can Agents Work. "Ambig-SWE: frontier results and sources." https://canagentswork.com/benchmarks/ambig-swe/ (accessed 2026-09-24). CC BY 4.0.@misc{caw-ambig-swe,
title = {{Ambig-SWE: frontier results and sources}},
author = {{Can Agents Work}},
year = {2026},
howpublished = {\url{https://canagentswork.com/benchmarks/ambig-swe/}},
note = {Accessed 2026-09-24. CC BY 4.0}
}