AWS Shipped 200 Cloud Services Written by Agents, and Published the Scorecard

On November 11, AWS released a local emulator of its own APIs. It covers more than 200 services, runs on a laptop or in CI, needs no AWS account, and is dedicated to the public domain under CC0.
It was also written, tested, and reviewed entirely by autonomous agents. Matt Goldberg, the senior principal engineer who led the project, says his team did not write or review a line of it.
That sentence is the release. Everything else is a developer tool.
What actually shipped
The emulator answers AWS API calls on a local endpoint, port 9297 by default, and holds resources in memory. Point the AWS CLI or an SDK at it with environment variables and your code runs unchanged. The build starts from 438 public Smithy API models, with one Rust crate per service. Unsupported operations return 501 rather than pretending.
Installation is RPM or Debian packages on Linux x86_64 and aarch64, Docker on macOS and Windows with a 2 GiB memory limit, or an embedded mode that compiles the emulator into Rust test binaries. Building from source wants 100 GB of free disk, so most people will use packages. There is an aws-emulator skill for coding agents and CLI support from AWS CLI 2.35.0.
The context matters here. LocalStack, the long-standing option for local AWS testing, moved its free edition behind an account in December 2025. A free, no-account, first-party alternative fills a hole that a lot of CI pipelines noticed.
The number AWS chose to publish
The release notes include a fidelity scorecard, and it is not flattering.
Of 42,857 scored tests, 24,243 pass. That is 56.6%. Step-level results are 49,810 of 68,007, and cross-service tests are 2,752 of 6,569. Per service it is wildly uneven: AWS KMS passes 291 of 310, or 93.9%. Amazon SQS passes 89.8%. CloudFormation passes 414 of 2,612, which is 15.8%.

Publishing that is the most interesting decision in the whole release. A company confident in a tool leads with the good number and buries the rest. AWS led with an average that includes a 15.8%, and put the failing behaviours in a folder in the repository called things-that-do-not-work.
Read the scores as a map rather than a verdict. KMS at 93.9% means the crypto API surface is trustworthy enough to write tests against. CloudFormation at 15.8% means anything involving stacks will diverge from real AWS in ways that will waste your afternoon. If your infrastructure code is CloudFormation-heavy, this is not your tool yet.
How agents built it
The origin is a weekend experiment. Goldberg had been watching agent loops burn 45 to 120 seconds per cycle deploying test stacks to a cloud sandbox just to check whether an SQS visibility timeout behaves. He gave Kiro, Amazon's spec-driven coding agent, a three-step prompt: write tests based on the AWS CLI, run them until they pass against real AWS, then build an in-memory version that passes the same tests. A rough SQS took about fifteen minutes. The rest of AWS followed over the weekend, and his team was testing against the prototype on Monday.
Turning that into a product took longer, and the reason is the part worth stealing. Agents with a free hand drift. An early version tried to build its own Python interpreter to run Lambda functions, because that is a locally coherent way to solve the problem it was given. Goldberg's metaphor is a colouring book: a child given a blank page and a pencil produces chaos, a child given outlined areas produces something coherent.
So AWS split the work into three bounded teams. Test writers turn API models into realistic scenarios and verify every scenario against real AWS before it counts. Issue finders replay the verified suite against the emulator and file each divergence, grouped by probable root cause. Fixers take one issue each, find the cause, and submit a patch.
Every patch then passes a merge gate: it must build, meet coding standards, survive multiple adversarial security reviews run by other agents, and cause zero fidelity regressions. Hundreds of agents work concurrently. AWS says they fix thousands of issues a day. The public history counted 29,684 commits on main at launch, and there were seventy numbered releases between November 8 and November 12.
Why nobody reviewed the code
The absence of human review is not a marketing position, it is a consequence of scale. Thirty thousand commits in four days cannot be reviewed by people. AWS replaced review with three things it could automate: a specification that already existed publicly, a reference implementation that could be queried, and a test suite whose authority came from having passed against the real cloud first.
Notice that none of those are model improvements. The recipe is spec, reference, verified tests, merge gate. Every team that owns an API surface already has the first two, which is why this is the transferable part of the announcement. You do not need AWS-scale resources to copy the structure. You need a way for the agent to know when it is wrong, and a gate that refuses to merge when it is.
The limits, stated by the vendor
No throttling simulation. No outage or region simulation. No sandboxing. Not for production use, not for security testing. Telemetry is on by default and is switched off with --telemetry off, the EMULATOR_TELEMETRY=off variable, or a line in config.toml. The repository does not accept pull requests, only bug and fidelity reports, which is an unusual stance for a CC0 project and probably the right one when the codebase is machine-generated at this rate.
Passing an emulator test is not passing in production. Permissions, quotas, IAM policy evaluation, and latency all differ, and the emulator's own docs say so. Anyone who points a CI pipeline at it and assumes green means correct will find out otherwise.
What this actually proves
The claim to be skeptical of is "agents wrote a cloud". They wrote 200 API-shaped surfaces that pass 56.6% of a test suite built from public API models. That is a genuine engineering result and a much narrower one than the headline suggests.
The claim worth taking seriously is about the scaffolding. When a specification is precise, a reference is queryable, and a gate is automatic, a swarm of agents can produce a large, partially correct, continuously improving artifact with no human in the review path. The output is honest about its own incompleteness because the scorecard ships with the release.
That combination, not the 200 services, is what other engineering organizations should be copying this quarter. Most of them have an internal API and no cheap way to test against it. The emulator is a demonstration that the cheap way can be built by the same agents that will then use it.
Related articles
Google replaced the model behind Search, and the number that moved was a shell benchmark
Gemini 4 Hydrogen became the default on 12 November, taking over from Gemini 3.8 Flash across Search, the Gemini app and the developer API. It runs about 2.5 times faster, scores 57% on Terminal-Bench 4.1 where its predecessor scored 18.2%, and costs $1.50 in and $7.50 out per million tokens.
Three ways a coding agent wastes money, measured on 30 repositories
TokenEconomizer watches what an agent did in a repository you work in often, then stops it from rediscovering the same build command, loading tools it never uses and looping on one error. Across five agent and model combinations it cut mean cost by 18.4% to 30.9%, cheaper in all 30 repository cells, with Pass@1 up in four of five.
earshot gives coding agents 922 sounds, and every one is rendered from code
An agent can write a game, an interface or a film and never hear any of it. earshot answers requests like "three variations of a footstep on gravel" from a catalogue of 922 designs built from public-domain recordings, with no audio files in the repository at all. It runs on CPU, as a CLI, a Python package, an MCP server and a Claude Code plugin.
The agents stealing API keys were not sophisticated. The gateways were open.
Amazon's decoy servers spent October posing as inference APIs and logged agents built on ARTEX pulling provider keys out of LiteLLM, Dify, Langflow and Open WebUI. The way in was a default master key and a half-finished install. ARTEX itself ships with budgets, scope gates and evidence for every finding.