Cloudflare Open-Sourced the Auditor It Runs on Itself. The Point Is That It Argues.

Cloudflare published a skill this week that turns a coding agent into a security auditor, and it became the fastest-growing repository on GitHub's weekly board — roughly 16,400 stars in seven days, 51,000 total. The company's own framing is unusually modest: this is the single-repo starting point that its internal vulnerability-discovery harness evolved from, released so other people can build their own.

What's worth reading isn't the attack-class catalogue, though that's extensive. It's the part about how findings get argued down.

Six phases, one of which is just bookkeeping

The workflow is structured rather than conversational, which is the first thing that distinguishes it from "ask Claude to find bugs."

Phase one is reconnaissance: map the architecture, the trust boundaries, the input surfaces, and — this is the unusual bit — write it to architecture.md and coverage-ledger.json. Phase two assigns isolated hunters to units from that ledger, records what each one checked, then runs coverage critics to find what nobody looked at. Phase three hands every unique candidate to a fresh verifier whose job is to disprove it. Phase four writes confirmed, needs_validation, or rejected records to findings.json and validates them against a JSON schema. Phase five has fresh agents verify the final source claims, and any material replacement gets another independent verifier. Phase six derives the reports from the verified records.

The two validators are plain Node scripts with no dependencies: validate-coverage-ledger.cjs runs after the ledger is created and after every update, validate-findings.cjs runs in phase four and again after every phase-five replacement. They're the least interesting files in the repo and the reason the rest of it holds together. An agent that can't produce a schema-valid finding can't claim one.

The three verdicts are the actual design

Most agent security tools have two outputs: a list of bugs, or nothing. This one has three, and the middle category is where the thinking is.

confirmed means a complete source trace plus a bounded, observed result. needs_validation means an exact unresolved fact and no severity assigned. rejected means a candidate that was actively disproven, recorded rather than deleted.

That third one is unusual and, I think, correct. A disproven candidate is evidence about the codebase — it tells the next run not to spend time there, and it tells a reviewer what was considered. Most tools throw that away.

The design principle underneath is stated bluntly in the README: only confirm established boundary failures, and if a source-grounded lead is blocked, keep it as needs_validation with its exact unresolved fact rather than upgrading it to a finding. There's a related rule that defense-in-depth gaps are not vulnerabilities — if layer A blocks the attack, the absence of layer B is a hardening note.

Anyone who has read a machine-generated vulnerability report knows exactly why those rules exist. The failure mode of LLM security review isn't missing bugs. It's confident noise: patterns that look like vulnerabilities, reported at high severity, wasting a triage team's week.

Adversarial validation, taken literally

The single most important line: the agent that checks a finding is never the agent that found it.

That's not a prompt instruction, it's an orchestration constraint, and it's the only way this kind of tool produces anything trustworthy. An LLM asked "is this finding correct?" about its own output will agree with itself. Give it a fresh context, a different agent instance, and the explicit task of disproving the claim, and you get something closer to a code review.

The phase-five step extends this: fresh agents verify the final source claims, and if a claim is materially replaced, another independent verifier checks the replacement. That's two levels of separation between the hunter and the record that survives.

The honest number

Buried in the design principles is this: in Cloudflare's test runs, a single run found roughly half of the vulnerabilities that repeated runs found in total.

Publishing that is a choice. It tells you the tool is a coverage machine, not an oracle, and it reframes the workflow — you don't run it once before a release, you run it repeatedly and let the ledgers accumulate. The repo explicitly supports this: multiple runs against the same repository are additive, using prior ledgers and findings to target gaps, revalidating changed source, and carrying forward current-source evidence without treating stale work as covered.

Half is also a reasonable estimate for human auditors on a first pass, which is either reassuring or damning depending on how you feel about the profession.

The requirement that will stop most people

The README lists a hard requirement that most users will skip past and then hit: an OS-enforced sandbox for builds, tests, processes, browsers, emulators, fuzzers, and fixtures, with external networking disabled, a sanitized allowlisted environment, enforced resource limits, and writes restricted to scratch paths.

Without those controls, the workflow keeps the lead at needs_validation instead of executing target code. That's the right call — you are, after all, running a language model's suggestions against a codebase — but it means the tool degrades into a static-analysis-plus-reasoning exercise for anyone without container infrastructure.

It also means this is not a laptop tool, despite the npx install line. The install is one command:

npx skills add https://github.com/cloudflare/security-audit-skill \
  --skill security-audit

And usage is just asking:

security audit this codebase

Which is a strange contrast: the invocation is a sentence, the prerequisites are a sandboxing policy. The gap between those two things is where most of the deployment effort will go.

What the output actually looks like

Three artifacts, derived rather than written: REPORT.md, FINDINGS-DETAIL.md, and NEEDS-VALIDATION.md, plus the underlying findings.json and the coverage ledger.

That split is more useful than it sounds. A single vulnerability report forces every finding into the same register, which is how you end up with a critical SQL injection and a missing CSP header adjacent in a list. Separating confirmed findings from open questions from derived summaries means the person reading NEEDS-VALIDATION.md knows they're looking at work, not results.

The default output directory is ~/security-audit-skill/<repo-name>/run-<N>, and the workflow only writes inside the target repository if you explicitly point it at a gitignored directory. Small detail, correct instinct: audit artifacts in a repo get committed by accident, and a committed list of unpatched vulnerabilities is its own incident.

How to tell whether it worked

Run it twice. That's the test the README implies without stating: because runs are additive and single runs find roughly half, the second run against the same commit is a direct measurement of the tool's coverage on your codebase. If the second run surfaces nothing new, either you're clean or the hunting classes don't match your stack.

Then check the rejection log. A run that confirms five issues and rejects forty candidates is doing the job. A run that confirms forty and rejects none has stopped verifying, and you should read those forty by hand before believing any of them.

What it signals about the category

Three things, briefly.

Agent skills are turning into distributions. This isn't a library you import; it's a folder of markdown and two validator scripts that a model reads at runtime. The repo is 20-odd files of instructions plus attack-class catalogues, split by target type — memory safety and binary, LLM and agent surfaces, web protocol and auth, client-side, supply chain, cloud and deployment, RPC and messaging, resource exhaustion, data isolation, desktop and local IPC. That structure is the product.

Verification is becoming the differentiator. Every vendor can generate a plausible finding now. Almost none can tell you which ones survived an adversarial pass, written to a schema, with the disproven ones preserved.

And the honest-number habit is spreading. A vendor that publishes "we found half" is a vendor whose next claim is more believable.

The repo is MIT-licensed, which means the interesting question isn't whether it works — it's whether your team can afford the sandbox it needs to work properly.