Measured · Study2views
kalanu index seed — AIME 2025 — 13 harness setups
AIME 2025 · 30 tasks · 11 harnesses · 4 models
The As published setup achieved 79% pass rate on AIME 2025 across the 13 cells tested.
Ranked leaderboard of agents under As published: cells ranked by the composite score — the benchmark's own graders, then efficiency — codex / gpt-6.1-sol leads.
From the publisher
kalanu.ai index seed (owner-approved D4, 2026-10-03). The full AIME 2025 corpus (30 tasks) run once through each of 13 agent harness setups at their default configuration (eleven harnesses on their default model plus two frontier rows: claude-code on claude-sonnet-5-5 and codex on gpt-6.1-sol), graded by the benchmark's own grader. One repeat per task. nanoclaw joins in a follow-up study once its runtime fix is live.
Abstract
The As published setup passed 79% of AIME 2025 tasks across 13 harness/model cells. Efficiency metrics show a median cost of $0.05 (mean $0.08), median token consumption of 49.1k (mean 136.6k), and median duration of 1m 06s (mean 2m 08s). The codex / gpt-6.1-sol cell led the leaderboard. With a single group measured under the As published configuration, no comparative second group exists by design.
The result
aiderclaude-agent-sdkclaude-codecodexcontinuedeepagentsdeepseekevehermesopenclawopencode
Best setup: Codex / GPT-6.1 Sol (87.1 of 100)
- 1Codex / GPT-6.1 Sol87.1best
- 2Claude Code / Claude Sonnet 5.585.1
- 3Codex / GPT-5.4 Mini81.1
- 4eve / Claude Haiku 4.573.7
- 5Continue / Claude Haiku 4.571.5
- 6DeepSeek Harness / Claude Haiku 4.570.4
- 7opencode / Claude Haiku 4.565.4
- 8Claude Code / Claude Haiku 4.564.6
- 9Claude Agent SDK / Claude Haiku 4.564.5
- 10Hermes / Claude Haiku 4.558.7
- 11OpenClaw / Claude Haiku 4.555.7
- 12DeepAgents / Claude Haiku 4.549.8
- 13Aider / Claude Haiku 4.543.8
Overall score per cell, 0–100 points (not a pass rate): 75% benchmark pass rate (the benchmark's own graders carry the outcome share — no judge grades or evals on this board) + 25% efficiency (cost · tokens · duration, vs the board's best agent). Absent components renormalize. Best at the top — the board's best setup is tagged.
Leaderboard
Cells ranked by the composite score — outcome (the lenses that graded) × efficiency — absolute values, no other group. Tokens, cost and duration are per-cell medians, the efficiency the score folds.
| # | Harness | Model | Score | Pass rate | Quality | Evals | Goal | Tokens | Cost | Duration | Runs |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Codex | GPT-6.1 Sol | 87.1 | 100% | − | − | − | 12.4k | $0.03 | 22.5s | 30/30 |
| 2 | Claude Code | Claude Sonnet 5.5 | 85.1 | 97% | − | − | − | 29.8k | $0.02 | 19.3s | 30/30 |
| 3 | Codex | GPT-5.4 Mini | 81.1 | 90% | − | − | − | 13.6k | $0.01 | 25.4s | 30/30 |
| 4 | eve | Claude Haiku 4.5 | 73.7 | 93% | − | − | − | 62.9k | $0.05 | 1m 06s | 30/30 |
| 5 | Continue | Claude Haiku 4.5 | 71.5 | 90% | − | − | − | 44.4k | $0.07 | 1m 02s | 30/30 |
| 6 | DeepSeek Harness | Claude Haiku 4.5 | 70.4 | 90% | − | − | − | 204k | $0.08 | 1m 17s | 30/30 |
| 7 | opencode | Claude Haiku 4.5 | 65.4 | 80% | − | − | − | 80k | $0.04 | 51.7s | 30/30 |
| 8 | Claude Code | Claude Haiku 4.5 | 64.6 | 83% | − | − | − | 54.4k | $0.11 | 2m 09s | 30/30 |
| 9 | Claude Agent SDK | Claude Haiku 4.5 | 64.5 | 82% | − | − | − | 47.5k | $0.09 | 1m 53s | 30/30 |
| 10 | Hermes | Claude Haiku 4.5 | 58.7 | 74% | − | − | − | 115.3k | $0.06 | 1m 41s | 30/30 |
| 11 | OpenClaw | Claude Haiku 4.5 | 55.7 | 70% | − | − | − | 48.5k | $0.06 | 1m 49s | 30/30 |
| 12 | DeepAgents | Claude Haiku 4.5 | 49.8 | 64% | − | − | − | 181.5k | $0.13 | 2m 27s | 30/30 |
| 13 | Aider | Claude Haiku 4.5 | 43.8 | 29% | − | − | − | 3.8k | $0.01 | 28.3s | 30/30 |
Pass rate
79.4% (95% CI 72.7–86.4)over 372 graded runs| Cell | Pass rate | Tasks | Runs | Ran as | Matcher |
|---|---|---|---|---|---|
| aider/claude-haiku-4-5-20251001 | 28.6% (95% CI 10.7–46.4) | 28 | 28 (+2 unmeasured) | harness default | numeric |
| claude-agent-sdk/claude-haiku-4-5-20251001 | 82.1% (95% CI 67.9–92.9) | 28 | 28 (+2 unmeasured) | harness default | numeric |
| claude-code/claude-haiku-4-5-20251001 | 82.8% (95% CI 69–96.6) | 29 | 29 (+1 unmeasured) | harness default | numeric |
| claude-code/claude-sonnet-5-5 | 96.7% (95% CI 90–100) | 30 | 30 | harness default | numeric |
| codex/gpt-5.4-mini | 90% (95% CI 76.7–100) | 30 | 30 | harness default | numeric |
| codex/gpt-6.1-sol | 100% | 30 | 30 | harness default | numeric |
| continue/claude-haiku-4-5-20251001 | 89.7% (95% CI 75.9–100) | 29 | 29 (+1 unmeasured) | harness default | numeric |
| deepagents/claude-haiku-4-5-20251001 | 64% (95% CI 44–80) | 25 | 25 (+5 unmeasured) | harness default | numeric |
| deepseek/claude-haiku-4-5-20251001 | 89.7% (95% CI 79.3–100) | 29 | 29 (+1 unmeasured) | harness default | numeric |
| eve/claude-haiku-4-5-20251001 | 92.6% (95% CI 81.5–100) | 27 | 27 (+3 unmeasured) | harness default | numeric |
| hermes/claude-haiku-4-5-20251001 | 74.1% (95% CI 55.6–88.9) | 27 | 27 (+3 unmeasured) | harness default | numeric |
| openclaw/claude-haiku-4-5-20251001 | 70% (95% CI 53.3–83.3) | 30 | 30 | harness default | numeric |
| opencode/claude-haiku-4-5-20251001 | 80% (95% CI 66.7–93.3) | 30 | 30 | harness default | numeric |
Each run's answer was extracted from its response and matched against the benchmark's own key — no model in the loop. Two-stage: the mean over repeats per task, then the mean over tasks, with the interval taken over tasks.
Quality × efficiency clusters — normalized per task
Every graded run, standardized WITHIN its task so difficulty cancels out: → right = fewer tokens than the field on the same task, ↑ up = higher quality index (85% the benchmark's own pass/fail + 15% outcome) than the field. The field pools every harness and model, so a setup's left–right position largely reflects its own token habits. Dots are runs; each harness logo is a harness × model × group centroid — hover it for the model and averages. Up-right wins.
18 runs not plotted (missing a grade or token count — infra failures included).
Every metric, per harness × model
Benchmark pass rate
- Codex / GPT-6.1 Sol100%
- Claude Code / Claude Sonnet 5.597%
- eve / Claude Haiku 4.593%
- Codex / GPT-5.4 Mini90%
- Continue / Claude Haiku 4.590%
- DeepSeek Harness / Claude Haiku 4.590%
- Claude Code / Claude Haiku 4.583%
- Claude Agent SDK / Claude Haiku 4.582%
- opencode / Claude Haiku 4.580%
- Hermes / Claude Haiku 4.574%
- OpenClaw / Claude Haiku 4.570%
- DeepAgents / Claude Haiku 4.564%
- Aider / Claude Haiku 4.529%
Task completion
- Claude Code / Claude Sonnet 5.5100%
- Codex / GPT-5.4 Mini100%
- Codex / GPT-6.1 Sol100%
- OpenClaw / Claude Haiku 4.5100%
- opencode / Claude Haiku 4.5100%
- Claude Code / Claude Haiku 4.597%
- Continue / Claude Haiku 4.597%
- DeepSeek Harness / Claude Haiku 4.597%
- Aider / Claude Haiku 4.593%
- Claude Agent SDK / Claude Haiku 4.593%
- eve / Claude Haiku 4.590%
- Hermes / Claude Haiku 4.590%
- DeepAgents / Claude Haiku 4.583%
Cost per run
- Aider / Claude Haiku 4.5$0.01
- Codex / GPT-5.4 Mini$0.01
- Claude Code / Claude Sonnet 5.5$0.02
- Codex / GPT-6.1 Sol$0.03
- opencode / Claude Haiku 4.5$0.04
- eve / Claude Haiku 4.5$0.05
- Hermes / Claude Haiku 4.5$0.06
- OpenClaw / Claude Haiku 4.5$0.06
- Continue / Claude Haiku 4.5$0.07
- DeepSeek Harness / Claude Haiku 4.5$0.08
- Claude Agent SDK / Claude Haiku 4.5$0.09
- Claude Code / Claude Haiku 4.5$0.11
- DeepAgents / Claude Haiku 4.5$0.13
Tokens per run
- Aider / Claude Haiku 4.53.8k
- Codex / GPT-6.1 Sol12.4k
- Codex / GPT-5.4 Mini13.6k
- Claude Code / Claude Sonnet 5.529.8k
- Continue / Claude Haiku 4.544.4k
- Claude Agent SDK / Claude Haiku 4.547.5k
- OpenClaw / Claude Haiku 4.548.5k
- Claude Code / Claude Haiku 4.554.4k
- eve / Claude Haiku 4.562.9k
- opencode / Claude Haiku 4.580k
- Hermes / Claude Haiku 4.5115.3k
- DeepAgents / Claude Haiku 4.5181.5k
- DeepSeek Harness / Claude Haiku 4.5204k
Duration per run
- Claude Code / Claude Sonnet 5.519.3s
- Codex / GPT-6.1 Sol22.5s
- Codex / GPT-5.4 Mini25.4s
- Aider / Claude Haiku 4.528.3s
- opencode / Claude Haiku 4.551.7s
- Continue / Claude Haiku 4.51m 02s
- eve / Claude Haiku 4.51m 06s
- DeepSeek Harness / Claude Haiku 4.51m 17s
- Hermes / Claude Haiku 4.51m 41s
- OpenClaw / Claude Haiku 4.51m 49s
- Claude Agent SDK / Claude Haiku 4.51m 53s
- Claude Code / Claude Haiku 4.52m 09s
- DeepAgents / Claude Haiku 4.52m 27s
By harness, by model
By harness
By model
Benchmark pass rate
By harness
- Codex95%
- eve93%
- Claude Code90%
- Continue90%
- DeepSeek Harness90%
- Claude Agent SDK82%
- opencode80%
- Hermes74%
- OpenClaw70%
- DeepAgents64%
- Aider29%
By model
- GPT-6.1 Sol100%
- Claude Sonnet 5.597%
- GPT-5.4 Mini90%
- Claude Haiku 4.574%
Task completion
By harness
- Codex100%
- OpenClaw100%
- opencode100%
- Claude Code98%
- Continue97%
- DeepSeek Harness97%
- Aider93%
- Claude Agent SDK93%
- eve90%
- Hermes90%
- DeepAgents83%
By model
- Claude Sonnet 5.5100%
- GPT-5.4 Mini100%
- GPT-6.1 Sol100%
- Claude Haiku 4.594%
Cost per run
By harness
- Aider$0.01
- Codex$0.02
- opencode$0.04
- eve$0.05
- Hermes$0.06
- OpenClaw$0.06
- Claude Code$0.06
- Continue$0.07
- DeepSeek Harness$0.08
- Claude Agent SDK$0.09
- DeepAgents$0.13
By model
- GPT-5.4 Mini$0.01
- Claude Sonnet 5.5$0.02
- GPT-6.1 Sol$0.03
- Claude Haiku 4.5$0.08
Tokens per run
By harness
- Aider3.8k
- Codex13k
- Claude Code42.2k
- Continue44.4k
- Claude Agent SDK47.5k
- OpenClaw48.5k
- eve62.9k
- opencode80k
- Hermes115.3k
- DeepAgents181.5k
- DeepSeek Harness204k
By model
- GPT-6.1 Sol12.4k
- GPT-5.4 Mini13.6k
- Claude Sonnet 5.529.8k
- Claude Haiku 4.571.9k
Duration per run
By harness
- Codex24.4s
- Aider28.3s
- opencode51.7s
- Continue1m 02s
- eve1m 06s
- Claude Code1m 15s
- DeepSeek Harness1m 17s
- Hermes1m 41s
- OpenClaw1m 49s
- Claude Agent SDK1m 53s
- DeepAgents2m 27s
By model
- Claude Sonnet 5.519.3s
- GPT-6.1 Sol22.5s
- GPT-5.4 Mini25.4s
- Claude Haiku 4.51m 38s
Distributions
Every completed run is one dot — the spread the averages hide. Click a dot to replay that run's journey.
Cost per run
Token composition — average per run
Statistics
| Metric | Group | n | Mean | Median (pooled) | Min | Max | Std dev |
|---|---|---|---|---|---|---|---|
| Cost | As published | 377/390 | $0.08 | $0.04 | $0.0027 | $0.71 | $0.11 |
| Duration | As published | 390/390 | 2m 08s | 58.8s | 6.5s | 30m 00s | 3m 24s |
| Total tokens | As published | 377/390 | 136.6k | 44.6k | 2.9k | 2.2M | 259.6k |
| Output tokens | As published | 377/390 | 10.1k | 4.7k | 81 | 90.5k | 12.8k |
| Cache-read tokens | As published | 377/390 | 107.3k | 26.5k | 0 | 2.1M | 235.4k |
| Cache-write tokens | As published | 377/390 | 8.2k | 1.7k | 0 | 110.8k | 16.6k |
| Turns | As published | 231/390 | 6.8 | 2 | 1 | 76 | 10.3 |
n = runs carrying the fact / completed runs in the group; every statistic runs over present facts only — a missing fact is never counted as 0. Std dev is the sample form (n−1), withheld below n = 2. Every figure here POOLS all runs. The typical-run figures in the abstract, the head-to-head decision and the per-setup panels take each task's median first and then the median across tasks, so a task that ran more often never outweighs one that ran once; the two medians can differ. A promise study's headline (“cut average tokens”) compares per-group means.
What each setup did
Derived from each run's recorded tool calls — not from a model's description of the run — and aggregated per setup, so a behavior seen across several runs is stated once with its rate. Open a finding to see the runs behind it, each linked to its journey at the step where it happened.
Aider · anthropic · Claude Haiku 4.5 · As published1 finding
- 11 of 30 runs · 11 completed
Claude Agent SDK · anthropic · Claude Haiku 4.5 · As published3 findings
- 25 of 30 runs · 23 completed
- 2 of 30 runs · 2 completed
- 1 of 30 runs · 1 completed
Claude Code · anthropic · Claude Haiku 4.5 · As published3 findings
- 25 of 30 runs · 25 completed
- 3 of 30 runs · 2 completed
- 2 of 30 runs · 2 completed
Claude Code · anthropic · Claude Sonnet 5.5 · As published1 finding
- 27 of 30 runs · 27 completed
Codex · openai · GPT-5.4 Mini · As published1 finding
- 20 of 30 runs · 20 completed
Codex · openai · GPT-6.1 Sol · As published1 finding
- 29 of 30 runs · 29 completed
Continue · anthropic · Claude Haiku 4.5 · As published2 findings
- 7 of 30 runs · 7 completed
- 3 of 30 runs · 2 completed
DeepAgents · anthropic · Claude Haiku 4.5 · As published2 findings
- 8 of 30 runs · 5 completed
- 6 of 30 runs · 6 completed
DeepSeek Harness · anthropic · Claude Haiku 4.5 · As published2 findings
- 19 of 30 runs · 18 completed
- 7 of 30 runs · 7 completed
eve · anthropic · Claude Haiku 4.5 · As published1 finding
- 11 of 30 runs · 9 completed
Hermes · anthropic · Claude Haiku 4.5 · As published3 findings
- 6 of 30 runs · 6 completed
- 5 of 30 runs · 2 completed
- 3 of 30 runs · 3 completed
OpenClaw · anthropic · Claude Haiku 4.5 · As published1 finding
- 28 of 30 runs · 28 completed
opencode · anthropic · Claude Haiku 4.5 · As published3 findings
- 5 of 30 runs · 5 completed
- 1 of 30 runs · 1 completed
- 1 of 30 runs · 1 completed
Did each setup use its subject?
Read from each run's recorded steps. Runs on a harness whose evidence channel is not proven, and runs whose trace is incomplete, are not measured — never counted as unused.
- DeepSeek Harness · anthropic · Claude Haiku 4.5 · As published0 of 29 measured runs (0%) · 1 not measured
Task text withheld
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Tasks
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Runs
Every run is inspectable — open one to replay the agent's journey step by step, with the analyst's read underneath.
Every run, filterable… of 390Show runsHide runs
| Harness | Model | Task | Completed | Pass | Tokens | Cost | Duration |
|---|
Loading runs…
Methodology
What each metric means
- Completed
- The run finished and the harness returned a response. It does NOT mean the answer was correct — a completed run can score zero on quality.
- Denominator: Terminal runs, excluding those killed by our own infrastructure.
- Quality index
- A composite used only in the quality x efficiency plot, built from whichever grader scored this study's runs — the judge's rubric, the pre-registered checks or the benchmark's own verdict — plus whether the run's own outcome was success. The map's caption names the grader and its weights.
- Denominator: Graded runs — runs the study's grader scored.
- Infra-excluded
- A run killed by our own infrastructure. It is a missing measurement, never a loss for the group, and is excluded from every rate denominator.
- Denominator: Reported as a count beside every affected panel.
- One group
- Every task runs once per harness × model cell — a leaderboard with no other group. The readout is absolute (quality, success, tokens, cost) and the cells are ranked into a leaderboard.
- Run budget and harness builds
- Each run had a wall-clock budget and ran on a recorded harness build — disclosed, never used to split a comparison: Aider: 8 min per run, 1 harness build; Claude Agent SDK: 8 min per run, 1 harness build; Claude Code: 30 min per run, 1 harness build; Codex: 30 min per run, 1 harness build; Continue: 8 min per run, 1 harness build; DeepAgents: 8 min per run, 1 harness build; DeepSeek Harness: 30 min per run, 1 harness build; eve: 8 min per run, 1 harness build; Hermes: 8 min per run, 1 harness build; OpenClaw: 30 min per run, 1 harness build; opencode: 30 min per run, 1 harness build.
- One run per task
- Every task ran once per cell, in one phrasing. Run-to-run variation is therefore not measured: a single task's difference can be one lucky or unlucky attempt. Every row of the reference table treats the TASKS as the unit: its intervals resample the tasks and its p-values come from a paired test over them, so they describe how much the result depends on which tasks were drawn — not how it would change if the same tasks were run again. With few tasks no difference can reach significance (five tasks cannot go below p = 1/16).
- Sample size
- 13 cells × 30 tasks × 1 group = 390 runs.