Method

How we build the atlas, and the rules behind every color, stripe, and label. When a rule changes, we say so on the timeline and in the change record below.

1. The territory: jobs

Jobs come from the SOC 2018 system through the O*NET 31.0 Database (997 civilian occupations; military occupations are out of scope because BLS OEWS does not cover them). We group occupations into 22 job families with ordered rules on SOC codes. Area on the atlas is US employment from the BLS OEWS May 2025 national estimates (830 detailed occupations). O*NET sub-occupations share the employment figure of their SOC group, so they change the pages, not the area.

2. Evidence: which benchmarks count

Each benchmark maps to one or more job families: primary for its main family and partial for others. Some benchmarks also map to specific occupations. Each benchmark also has a role:

Work: it tests whether agents complete the work of the families it maps to. A work benchmark mapped primary to a family is direct evidence for that family.
Broad: it spans many occupations, so it cannot speak for one family. Context only: APEX-Agents, BrowseComp, GAIA, GDPval, GDPval-AA, MCP-Atlas, MCPMark, METR Time Horizon, RLI, TheAgentCompany.
Behavior: it tests how agents behave (for example cheating on impossible tasks, asking questions, or single tutoring moves), not completed work. Context only: Ambig-SWE, ImpossibleBench, TutorMoments.

A retired benchmark stops counting on the date its maintainers or a major evaluator retired it. A field study (for example a company's safety record) measures real deployments; it adds to the work measured, but it gives no task success rate, so it cannot set a step.

3. The work ladder (color)

The color of a family is its step on the work ladder: the largest unit of work that direct evidence shows agents complete. We compute it by these rules, not by judgment. The step is a floor of what agents have shown on public benchmarks. It is not a forecast, and it does not say that agents can do a whole job.

Step	Rule
Most projects	Agents complete 60% or more of whole projects on two direct benchmarks from independent institutions.
Some projects	Agents complete 20% or more of whole projects on two direct benchmarks from independent institutions.
Nearly all tasks	Agents complete 90% or more of the tasks (or projects) on two direct benchmarks from independent institutions.
Most tasks	Agents complete 60% or more of the tasks (or projects) on two direct benchmarks from independent institutions.
Some tasks	Agents complete 20% or more of the tasks (or projects) on a direct benchmark.
Not yet	Direct benchmarks exist, but agents complete less than 20% of their tasks.
Unmeasured	No direct benchmark gives a success rate.

Success rate: the share of whole tasks or projects that the best agent completes. For a benchmark that reports the share of tasks completed, it is the frontier result against the ceiling (for example 64% of 100%). A benchmark that its maintainers or a major evaluator declared solved counts as 100%, when its metric counts completed tasks.
Partial credit: a rubric score, progress through a task, or a percentile is not a success rate. When the source also publishes a completion rate (for example "All-Pass", "Pass@1", or the share of perfect tasks), we use the best one. When it does not, the benchmark adds to the work measured but cannot set a step.
No success rate: Elo scores, money, time, and rates of other events (merges, crashes) cannot set a step.
Independent institutions: steps 3 to 6 need two direct benchmarks whose institutions do not overlap, so that one team's test design cannot decide a family alone.
Tasks and projects: a task is bounded, with a clear spec, from minutes to hours of human time. A project is a whole deliverable judged by an acceptance standard, from hours to days. A project benchmark also counts at the task steps (2 to 4), because a whole project is a larger unit of work. The project steps (5 and 6) use project benchmarks only.
Evidence that disagrees: a direct benchmark where agents complete less than 20% does not lower the step, but the family page shows it next to the evidence that sets the step.

4. How much of the work is measured (stripes)

A step on one benchmark can rest on a small part of a job. So the texture shows coverage: the share of a family's work that direct benchmarks measure.

The work: O*NET 31.0 links each task of an occupation to detailed work activities, and each of those belongs to one of 332 intermediate work activities (IWAs). We weight each task by its importance times its relevance (the share of workers who do it), split the weight equally over the task's IWAs, and scale each occupation's weights to 100%. A family's mix is the average over its occupations, weighted by US employment.
What a benchmark measures: we list the IWAs that its tasks make the agent do and that its grading checks. Each benchmark page shows them. A benchmark mapped to specific occupations measures only those occupations.
Coverage is the share of the family's mix that at least one direct benchmark measures. Solid: 50% or more. Thin stripes: 20% to 50%. Wide stripes: less than 20%.
O*NET has no work-activity data for occupations that hold 3% of US jobs. They count as not measured. Detailed O*NET codes are averaged into their SOC code, because BLS reports employment only by SOC code.

5. The last 12 months (arrow)

The data date is the latest date on which we retrieved a result (10 Oct 2026). We compute every step again with only the results dated on or before the same day one year earlier (10 Oct 2025) and the benchmarks that existed then. An up arrow means a higher step now, or a first step since then. A down arrow means a lower step, for example after a benchmark was retired.

6. The key

Color: the work ladder

Most projects
Agents complete 60% or more of whole projects on two direct benchmarks from independent institutions.
Some projects
Agents complete 20% or more of whole projects on two direct benchmarks from independent institutions.
Nearly all tasks
Agents complete 90% or more of the tasks (or projects) on two direct benchmarks from independent institutions.
Most tasks
Agents complete 60% or more of the tasks (or projects) on two direct benchmarks from independent institutions.
Some tasks
Agents complete 20% or more of the tasks (or projects) on a direct benchmark.
Not yet
Direct benchmarks exist, but agents complete less than 20% of their tasks.
Unmeasured
No direct benchmark gives a success rate.

Texture: how much of the work is measured

Most of the work is measured (50% or more)
Some of the work is measured (20% to 50%)
Little of the work is measured (less than 20%)

Coverage is the share of a job family's work, by O*NET work activities, that direct benchmarks test. How we measure it.

Arrow: the last 12 months

A higher step than on 10 Oct 2025, or measured for the first time since then.

A lower step than on 10 Oct 2025, for example after a benchmark was retired.

7. Benchmark status

Each benchmark page also shows a status. It compares the frontier result with a reference point that each benchmark declares. A ceiling is a hard maximum, such as 100% of tasks. A parity line is human-expert performance, such as 50% wins and ties against experts. The status describes the benchmark, not the job family.

Ceiling: open below 20%, emerging from 20%, strong from 60%, saturated from 90% of the ceiling.
Parity: open below 50% of parity, emerging from 50%, strong at or above parity.
No reference point (Elo scores, money balances, field signals, and lower-is-better behavior rates): unrated.
Solved and retired come from the maintainers or a major evaluator, with a source and a date.

Saturated: The best result is at 90% or more of the ceiling.
Strong: The best result is at 60–90% of the ceiling, or at or above human parity.
Emerging: The best result is at 20–60% of the ceiling, or at 50–100% of human parity.
Open: The best result is below 20% of the ceiling, or below half of human parity.
Unrated: No fixed reference point (for example Elo scores or field signals).
Solved: Its maintainers or a major evaluator declared it solved.
Retired: Maintainers or a major user stopped using it as a frontier measure.
No results yet: We have not recorded a result yet.

8. Readings

A reading is our own short text for a job family. It explains the step and the coverage in plain words and says what the evidence leaves out. It names its evidence and its date. It cannot change the step. The founder reviews every reading before it goes live, and the text carries a note when the step has changed since it was written.

9. Sources and provenance

Primary sources first: the benchmark's paper, site, repo, or leaderboard, or its maintainers' posts.
Lab-reported results come from a model or agent developer about its own system. We label them.
Aggregators help us discover results. We do not cite them as the source of a number.
Every result records its source URL, source kind, who reported it, and when we retrieved it. The build rejects a result without them.
When sources disagree, we show the primary value and log the disagreement in public. See conflicts.
We show the frontier and a few history points, and we link to the full leaderboard at the source.

10. What we do not do (yet)

No composite score. A step comes from named benchmarks by fixed rules, not from an average.
No runs of our own. Every result comes from its source. We plan independent runs later, starting with DevOps.
No logos without written permission. Names identify sources and do not imply endorsement.

11. Known limits

Benchmarks measure tasks, not jobs. A job is a bundle of tasks, judgment, and relationships.
Capability is not adoption. See usage research such as the Anthropic Economic Index for adoption.
Coverage is biased toward work that is easy to benchmark, especially software.
The work-activity mapping is our judgment, and O*NET describes work at a coarse level. A measured activity can still hide untested parts of the work.
Labor data is for the United States. Most benchmarks are in English.
Most results are single runs. Reliability over many runs is often unknown.

12. Changes to the method

24 Sep 2026, method 2: the work ladder (color) and coverage (stripes) replace the four written levels (Unexplored, Not yet, Partial, Strong) at two grains. Steps now follow fixed rules from direct evidence. Broad and behavior benchmarks became context only. Readings became short texts that explain the step.
23 Sep 2026, method 1: first release, with written levels for each family at the task and project grains.

13. Corrections

If a number, summary, or mapping is wrong, tell us. We aim to correct errors within 48 hours of a report and record the change. Every entry shows the date we last checked it against its sources.