Can Agents Work?

Jobs / Software engineering / 15-1253.00

Software Quality Assurance Analysts and Testers

Develop and execute software tests to identify software problems and their causes. Test system modifications to prepare for implementation. Document software and application defects using a bug tracking system and report defects to software or web developers. Create and maintain databases of known defects. May participate in software design reviews to provide input on functional requirements, operational characteristics, product designs, and schedules.

US jobs
186,740
Median wage
$104,300 a year
Job family
Software engineering

Can agents do the work of software quality assurance analysts and testers?

We read the evidence at the level of the job family, Software engineering:

  • Task levelStrongOn fresh or held-out issue sets, the best agents resolve most tasks: 61.5% on SWE-Bench Pro (public set) and 64.5% on SWE-rebench. Limits: the SWE-Bench Pro private set is lower (51.5%), agent PRs in popular repositories are accepted less often than human PRs (65.3% vs 76.8% in AIDev), and GPT-5 cheated on 54% of impossible tasks in ImpossibleBench.
  • Project levelNot yetWhole projects are mostly out of reach. On the Remote Labor Index, the best agent completed 15.8% of paid freelance projects to a standard a client would accept. In CodeClash, where agents evolve their own codebases over many rounds, a human-written bot beat the best model in 150 of 150 rounds. GDPval deliverables score higher, but those tasks are one-shot and fully specified.

Benchmarks for the job family

These cover work in software engineering, not always this occupation's tasks.

Typical tasks

From O*NET 31.0. Core tasks first. Few of these tasks have a benchmark today.

  • Identify, analyze, and document problems with program function, output, online screen, or content.
  • Document software defects, using a bug tracking system, and report defects to software developers.
  • Develop testing programs that address areas such as database impacts, software scenarios, regression testing, negative testing, error or bug retests, or usability.
  • Design test plans, scenarios, scripts, or procedures.
  • Document test procedures to ensure replicability and compliance with standards.
  • Provide feedback and recommendations to developers on software usability and functionality.
  • Install, maintain, or use software testing programs.
  • Test system modifications to prepare for implementation.
  • Create or maintain databases of known test defects.
  • Develop or specify standards, methods, or procedures to determine product quality or release readiness.
  • Monitor bug resolution efforts and track successes.
  • Update automated test scripts to ensure currency.
Other job titles for this occupation (68)

Automation Tester · Information Technology Analyst (IT Analyst) · Quality Assurance Analyst (QA Analyst) · Quality Assurance Engineer (QA Engineer) · Quality Engineer · Software Quality Assurance Analyst (SQA Analyst) · Software Quality Assurance Engineer (SQA Engineer) · Software Quality Engineer · Software Test Engineer · Test Engineer · Application Integrator · Applications Analyst · Applications Quality Assurance Specialist (Applications QA Specialist) · Applications Software Engineering Information Technology Specialist (Applications Software Engineering IT Specialist) · Applications System Analyst · Applications Tester · Automation Specialist · Beta Tester · Bug Bounty Hunter · Computer Consultant · Data Modeler · Game Tester · Implementation Specialist · Information Technology Consultant (IT Consultant) · IT Performance Analyst (Information Technology Performance Analyst) · Malware Analyst · Performance Tester · Quality Analyst · Quality Assurance Automation Engineer (QA Automation Engineer) · Quality Assurance Specialist (QA Specialist) · Quality Assurance Tester (QA Tester) · Secure Software Assessor · Server Engineer · Software Application Quality Assurance Specialist (Software Application QA Specialist) · Software Application Support Specialist · Software Development Test Engineer · Software Installer · Software Quality Assurance Consultant (Software QA Consultant) · Software Quality Assurance Specialist (SQA Specialist) · Software Quality Assurance Technician (SQA Technician) · Software Quality Control Specialist (Software QC Specialist) · Software Quality Tester · Software Requirements Engineer · Software Systems Engineer · Software Test Analyst · Software Tester · Software Testing Specialist · Solution Architect · Solutions Architect · System Testing and Evaluation Specialist · Systems Analyst · Systems Analyst Engineer · Systems Coordinator · Systems Integration Engineer · Systems Software Quality Assurance Specialist (Systems Software QA Specialist) · Systems Software Specialist · Systems Support Engineer · Systems Tester · Systems Tester Analyst · Technical Support Engineer · Technology Analyst · Test Automation Engineer · Usability Engineer · User Experience Designer (UX Designer) · User Interface and User Experience Architect (UI/UX Architect) · User Interface and User Experience Designer (UI/UX Designer) · User Interface Designer (UI Designer) · Validation Engineer