How we benchmark AI coding agents

One complete product brief. One autonomous run. Zero human semantic follow-up. Then we check whether the result could actually ship, and publish everything, including the runs that fail.

1. What we test: the whole stack, not a model name

A model behaves differently inside different agents, settings and permissions. So a 1P1A result is never "Claude" or "GPT". It is always:

model × model configuration × agent/harness version × tool permissions × environment × task version × run number

For example: Claude Code 2.1.252 + claude-sonnet-5, default effort, --dangerously-skip-permissions, macOS, game-micro-tower-defense-v0, run 1. When a CLI does not expose a version or setting, we write not recorded or unknown. We never guess.

2. The run rules

  1. One prompt, but a full spec. One prompt means one complete product brief, which can be long. It does not mean one short sentence.
  2. The agent works alone. It plans, codes, runs, tests and fixes inside its own harness, headless, with tool approval turned on automatically.
  3. No second prompt. Nobody types "fix the login" or "make it prettier". Any human semantic follow-up invalidates the run, or is recorded as an intervention.
  4. Self-correction is allowed. The agent may retry and fix itself as often as its harness allows. That cost shows up in time and tokens.
  5. Specs are frozen. A published spec never changes. A change means a new task version, for example v0 to v1, so old results stay comparable.

3. What we record for every run

Visual quality and community preference can be scored separately. They are never folded into the ship verdict.

4. How verdicts are made

Today: a person checks each requirement in the spec by hand, plays or uses the app, and writes the verdict into the run manifest. We also re-run simple checks in headless Chrome: console errors on load, and whether the page stays responsive once the app is in use.

Next: every canonical benchmark ships with an automated test suite that decides ship or no-ship with no human judgement, so anyone can re-run it.

5. One run is content. Three runs start to be a benchmark.

Agents are not deterministic. The same stack can ship on one run and fail on the next. So we only make a reliability claim with at least 3 runs per agent stack on the same frozen task, and we publish the spread, not just the best run. Every result on this site today is n = 1.

6. How we count cost

We publish what each CLI reports. Claude Code returns a list-price dollar figure (total_cost_usd). Codex CLI and Antigravity CLI return tokens only on subscription billing, so their cost is shown as unknown. We do not multiply tokens by a price we cannot verify. A normalized cost (tokens × each vendor's published list price, computed the same way for every stack) is planned, and will be labelled as computed by us.

7. How we pick tasks

The two game benchmarks on this site are deliberately lighter. They are a fun first look, not the canonical test.

8. Corrections

Run records are append-only. A wrong result gets a correction record next to it; the original is never edited or deleted. On Oct 31, 2026 we found that the Claude Code kart racing build on this site had been patched by one line before publishing, without disclosure. The unedited build freezes mid-race, so that run is now no-ship. Read the correction.

9. Neutrality

10. Data license

Run data on this site, including runs.json and llms-full.txt, is published under CC BY 4.0. Reuse it freely, credit 1P1A with a link to https://1p1a.com/.

FAQ

Which AI coding agent is best?

We do not name a winner yet. With one run per stack, the honest answer is: it depends on the task, and we need repeated runs to say more. See what each one did in the side-by-side comparison.

Can I suggest a task or an agent to test?

Yes. Send it to @tomcandev on X. Tasks that look like real work and can be tested automatically go to the front of the queue.

Will you rerun when new models come out?

Yes. Re-running the same frozen spec on new releases is the point. Next up are the September and October 2026 releases, including Claude Opus 5.5 and Sonnet 5.5, GPT-6.1 Sol in Codex CLI and Gemini 4 Argon in Antigravity CLI.

Last updated Oct 31, 2026.