How we benchmark AI coding agents
One complete product brief. One autonomous run. Zero human semantic follow-up. Then we check whether the result could actually ship, and publish everything, including the runs that fail.
1. What we test: the whole stack, not a model name
A model behaves differently inside different agents, settings and permissions. So a 1P1A result is never "Claude" or "GPT". It is always:
model × model configuration × agent/harness version × tool permissions × environment × task version × run number
For example: Claude Code 2.1.252 + claude-sonnet-5, default effort, --dangerously-skip-permissions, macOS, game-micro-tower-defense-v0, run 1. When a CLI does not expose a version or setting, we write not recorded or unknown. We never guess.
2. The run rules
- One prompt, but a full spec. One prompt means one complete product brief, which can be long. It does not mean one short sentence.
- The agent works alone. It plans, codes, runs, tests and fixes inside its own harness, headless, with tool approval turned on automatically.
- No second prompt. Nobody types "fix the login" or "make it prettier". Any human semantic follow-up invalidates the run, or is recorded as an intervention.
- Self-correction is allowed. The agent may retry and fix itself as often as its harness allows. That cost shows up in time and tokens.
- Specs are frozen. A published spec never changes. A change means a new task version, for example
v0tov1, so old results stay comparable.
3. What we record for every run
- Ship or no-ship verdict
- Automated tests passed out of total (from the first business-app benchmark onward)
- Wall-clock time, measured around the CLI call
- Input, cached and output tokens, as each CLI reports them
- Cost, only when the CLI reports it
- Human interventions (target: 0)
- Errors and failure mode
- The session log, the generated files and a playable or recorded result
- The full stack metadata from section 1
Visual quality and community preference can be scored separately. They are never folded into the ship verdict.
4. How verdicts are made
Today: a person checks each requirement in the spec by hand, plays or uses the app, and writes the verdict into the run manifest. We also re-run simple checks in headless Chrome: console errors on load, and whether the page stays responsive once the app is in use.
Next: every canonical benchmark ships with an automated test suite that decides ship or no-ship with no human judgement, so anyone can re-run it.
5. One run is content. Three runs start to be a benchmark.
Agents are not deterministic. The same stack can ship on one run and fail on the next. So we only make a reliability claim with at least 3 runs per agent stack on the same frozen task, and we publish the spread, not just the best run. Every result on this site today is n = 1.
6. How we count cost
We publish what each CLI reports. Claude Code returns a list-price dollar figure (total_cost_usd). Codex CLI and Antigravity CLI return tokens only on subscription billing, so their cost is shown as unknown. We do not multiply tokens by a price we cannot verify. A normalized cost (tokens × each vendor's published list price, computed the same way for every stack) is planned, and will be labelled as computed by us.
7. How we pick tasks
- Too easy is worthless: no todo apps, calculators or landing pages as canonical benchmarks.
- Too big is unrunnable: no full Salesforce clone.
- Sweet spot: about 20 to 40 minutes of frontier-agent work, 10 to 20 core requirements, a real business workflow, verifiable by automated tests. Candidates: a mini CRM with roles and audit history, a two-sided booking app, a realtime operations board.
The two game benchmarks on this site are deliberately lighter. They are a fun first look, not the canonical test.
8. Corrections
Run records are append-only. A wrong result gets a correction record next to it; the original is never edited or deleted. On Oct 31, 2026 we found that the Claude Code kart racing build on this site had been patched by one line before publishing, without disclosure. The unedited build freezes mid-race, so that run is now no-ship. Read the correction.
9. Neutrality
- No sponsorship, affiliate deal or promotion from any vendor whose agent appears in our results, if it could bend the numbers.
- We publish failures, including from the agents we use ourselves.
- No claim like "the model got smarter overnight" unless a version or backend change is verified.
10. Data license
Run data on this site, including runs.json and llms-full.txt, is published under CC BY 4.0. Reuse it freely, credit 1P1A with a link to https://1p1a.com/.
FAQ
Which AI coding agent is best?
We do not name a winner yet. With one run per stack, the honest answer is: it depends on the task, and we need repeated runs to say more. See what each one did in the side-by-side comparison.
Can I suggest a task or an agent to test?
Yes. Send it to @tomcandev on X. Tasks that look like real work and can be tested automatically go to the front of the queue.
Will you rerun when new models come out?
Yes. Re-running the same frozen spec on new releases is the point. Next up are the September and October 2026 releases, including Claude Opus 5.5 and Sonnet 5.5, GPT-6.1 Sol in Codex CLI and Gemini 4 Argon in Antigravity CLI.
Last updated Oct 31, 2026.