Benchmarks / IaC-Eval
IaC-Eval
Built by University of Michigan and Cisco Research · released 2024
A code-generation benchmark for Terraform. Each of 458 human-written scenarios pairs a natural-language request for AWS infrastructure with an intent specification. A model's Terraform program passes only if it plans successfully and meets the specified intent.
Frontier
36.7%
pass@1
GPT-4 with retrieval-augmented generation · harness: RAG over Terraform documentation
2024 · Source: University of Michigan (benchmark maintainers)
Best result across the paper's enhancement strategies (few-shot, chain of thought, multi-turn, RAG). Multi-turn GPT-4 scores 31.12%.
The best result is at 20–60% of the ceiling, or at 50–100% of human parity.
What it measures
Whether a model can write a correct Terraform configuration from a plain-English request. Generated programs are checked with Terraform tooling and Open Policy Agent rules that encode the scenario's required resources and settings. The headline is pass@1 accuracy over the 458 scenarios, estimated from 20 samples per problem.
pass@1: Share of scenarios where a single generated Terraform program plans and satisfies the intent specification. Higher is better.
Status compares the frontier with a ceiling of 100%.
Facts
- Grain
- Task level: bounded tasks with a clear spec
- Environment
- cli
- Grading
- automated-tests, state-check
- Tasks
- 458
- Reuse
- CC-BY-4.0 (Hugging Face dataset card) (open-cc-by)
Limits to keep in mind
- Results in the paper cover models from 2024 (GPT-4, GPT-3.5, Gemini 1.0 Pro, CodeLlama, WizardCoder, Magicoder). The README lists the leaderboard as "TBD" and no newer results are published by the maintainers. Source
- AWS and Terraform only. Other clouds and IaC tools are left as future work. Source
- The maintainers describe the framework as not production-ready and under active patching. Source
- A January 2025 update of the dataset (now labeled v1.1) fixed bugs in the difficulty 4 to 6 cases and rewrote them; it still has 458 questions. The paper's results use the original 2024 release. Source
Recorded results
A selection that shows the frontier over time. The full leaderboard is at the source.
| System | pass@1 | Date | Source |
|---|---|---|---|
| WizardCoder-33B-V1.1 | 8.93% | 2024 | University of Michigan · primary |
| GPT-4 (zero-shot) | 19.4% | 2024 | University of Michigan · primary |
| GPT-4 with retrieval-augmented generation · frontier harness: RAG over Terraform documentation | 36.7% | 2024 | University of Michigan · primary |
Timeline
- 14 Jan 2025 — IaC-Eval dataset update fixes the hardest cases and keeps all 458 questions. Source
Where it sits in the atlas
Work ladder: Direct evidence for DevOps, SRE, and IT operations. On the work ladder it counts as 36.7% of tasks completed. How the ladder works
Work it measures (O*NET work activities): Set up computer systems, networks, or other information systems; Program computer systems or production equipment.
Go to the source
Last checked 10 Oct 2026 against 8 primary sources. See an error? Tell us.
How to cite
Credit the original work first: IaC-Eval by University of Michigan and Cisco Research (https://proceedings.neurips.cc/paper_files/paper/2024/hash/f26b29298ae8acd94bd7e839688e329b-Abstract-Datasets_and_Benchmarks_Track.html).
Then, if you used this page:
Can Agents Work. "IaC-Eval: frontier results and sources." https://canagentswork.com/benchmarks/iac-eval/ (accessed 2026-10-11). CC BY 4.0.@misc{caw-iac-eval,
title = {{IaC-Eval: frontier results and sources}},
author = {{Can Agents Work}},
year = {2026},
howpublished = {\url{https://canagentswork.com/benchmarks/iac-eval/}},
note = {Accessed 2026-10-11. CC BY 4.0}
}