Software Quality Assurance Analysts and Testers

Benchmarks for the job family

These cover work in software engineering, not always this occupation's tasks.

Direct evidence

Work benchmarks mapped primary to this family. They set the step when they report how many tasks agents complete.

• Aider Polyglot leaderboardby AiderTask level · Percent correct

  88%gpt-5 (high) · 23 Aug 2025

  Strong
• AIDevby Queen's UniversityTask level · PR acceptance rate (AIDev-pop)

  65.3%OpenAI Codex · 20 Jul 2025

  Unrated
• Code Review Benchby MartianTask level · F2 score (Core profile, offline set)

  65.1Qodo Extended · 20 Aug 2026

  Strong
• CodeClashby Stanford University and Princeton UniversityProject level · Elo (all arenas)

  1385 EloClaude Sonnet 4.5 · 3 Nov 2025

  Unrated
• PR Arenaby aavetisTask level · Merge rate of ready PRs

  96%Copilot · 10 Oct 2026

  Unrated
• SWE-Bench Pro V2by Scale AI and ReflectionTask level · Resolve rate (V2 Full)

  99.4%Opus 5 (Claude Code) xhigh · 22 Sep 2026

  Saturated
• SWE-Lancerby OpenAITask level · IC SWE Diamond pass@1

  80%gpt-5.1-codex-max · 18 Nov 2025

  Strong
• SWE-rebenchby NebiusTask level · Resolved rate

  64.5%Anthropic Fable 5 [high] · 1 Jul 2026

  Strong
• Terminal-Bench 4.0by Laude Institute and Stanford UniversityTask level · Resolution rate

  64.8%Claude Code + Opus 5.5 (max) · 6 Oct 2026

  Strong

Also relevant

These benchmarks cover part of this family's work, or have no result yet. They do not set its step.

• BIRD-Interactby HKU and Google CloudTask level · Success rate, a-Interact, Full, both phases

  22.7%MERIT + GPT-5.4 (GPT-4o user simulator) · 22 May 2026

  Emerging
• CyberBench v1.1by Vals AITask level · Overall accuracy (mean of PoC and Patch pass rates)

  78%GPT-6 Sol · 7 Oct 2026

  Strong
• CyberGymby UC BerkeleyTask level · Success rate (Level 1)

  99.2%Lyrie Agent (multi-model) · 15 Sep 2026

  Saturated
• DARPA AI Cyber Challenge, Final Competitionby DARPA and ARPA-HProject level · Synthetic vulnerabilities found (scored round)

  43Team Atlanta · 8 Aug 2025

  Strong
• HWE-Benchby Peking UniversityTask level · Resolved rate

  79.9%GPT-5.6 Sol max (Codex CLI) · 31 Jul 2026

  Strong
• Spider 2.0by HKU and Salesforce AI ResearchTask level · Execution accuracy (Spider 2.0-Snow)

  96.7%Genloop's Sentinel Agent v2 Pro · 1 Mar 2026

  Saturated
• Terminal-Bench 2.0by Laude Institute and Stanford UniversityTask level · Resolution rate

  84.7%NexAU-AHE + GPT-5.5 · 23 Apr 2026

  Retired
• Will It Survive? Agent code survival studyby Concordia UniversityTask level · Line-level modification (death) rate

  No result recorded

  No results yet

Context only

These benchmarks span many occupations, or they test how agents behave. They give context but do not set the step.

• Ambig-SWEby Carnegie Mellon UniversityTask level · Underspecificity detection accuracy (best prompt)

  89%Claude Sonnet 4 · 21 Feb 2026

  Strong
• GDPvalby OpenAIProject level · Wins plus ties against experts

  84.9%GPT-5.5 · Apr 2026

  Strong
• GDPval-AAby Artificial AnalysisProject level · GDPval-AA Elo

  1866 EloClaude Opus 5.5 (Max, Default Fallback) · 10 Oct 2026

  Unrated
• ImpossibleBenchby Carnegie Mellon University and AnthropicTask level · Cheating rate (Conflicting-SWEbench, full scaffold)

  54%GPT-5 · 23 Oct 2025

  Unrated
• MCP Atlasby Scale AITask level · Pass rate

  88.1%Muse Spark 1.1 · 9 Jul 2026

  Strong
• MCPMark Verifiedby EVAL SYS, LobeHub and NUS TRAILTask level · Pass@1 (single run)

  96.1%kimi-k3-max · 20 Jul 2026

  Saturated
• METR Task-Completion Time Horizonsby METRTask level · 50% task-completion time horizon

  17 hClaude Mythos Preview (early) · 7 Apr 2026

  Unrated
• Remote Labor Indexby CAIS and Scale AIProject level · Automation rate

  21.3%Opus 5.5 · 7 Oct 2026

  Emerging
• TheAgentCompanyby Carnegie Mellon UniversityTask level · Tasks resolved

  42.9%TTE-MatrixAgent + DeepSeek-V3.2 · 10 Nov 2025

  Emerging

Retired

Their maintainers or a major evaluator stopped using them as a frontier measure. They do not count.

• SWE-Bench Proby Scale AITask level · Resolve rate (public set)

  61.5%Muse Spark 1.1 · 9 Jul 2026

  Retired
• SWE-bench Verifiedby Princeton University and OpenAITask level · Resolved

  79.2%Sonar Foundation Agent + Claude 4.5 Opus · 5 Dec 2025

  Retired

Typical tasks

From O*NET 31.0. Core tasks first. Few of these tasks have a benchmark today.

• Identify, analyze, and document problems with program function, output, online screen, or content.
• Document software defects, using a bug tracking system, and report defects to software developers.
• Develop testing programs that address areas such as database impacts, software scenarios, regression testing, negative testing, error or bug retests, or usability.
• Design test plans, scenarios, scripts, or procedures.
• Document test procedures to ensure replicability and compliance with standards.
• Provide feedback and recommendations to developers on software usability and functionality.
• Install, maintain, or use software testing programs.
• Test system modifications to prepare for implementation.
• Create or maintain databases of known test defects.
• Develop or specify standards, methods, or procedures to determine product quality or release readiness.
• Monitor bug resolution efforts and track successes.
• Update automated test scripts to ensure currency.