kalanuStudiesLeaderboard

As published: as published (nothing added)

Measured · Study2views

kalanu index seed — benchme — 13 harness setups

benchme — intent tasks · 163 tasks · 11 harnesses · 4 models

Deepseek / claude-haiku-4-5-20251001 led the As published leaderboard on the benchmark's pass rate.

Ranked leaderboard of agents under As published: cells ranked by the composite score — the benchmark's own graders, then efficiency — deepseek / claude-haiku-4-5-20251001 leads.

From the publisher

kalanu.ai index seed (owner-approved T1 tier, 2026-10-01). The full benchme corpus run once through each of 13 agent harnesses at their default configuration (twelve harnesses on their default model plus two frontier rows), graded by the benchmark's own reference answers or verifier. One repeat per task. nanoclaw joins in a follow-up study once its runtime fix is live.

How to read two rows. deepagents' pass rate (about 0.6%) measures deepagents with kalanu's default virtual backend, which gives the agent no shell and no HTTP tool, so it cannot reach the benchme apps at all. aider's pass rate (about 2.5%) measures one-shot aider, which edits files and cannot navigate a web application. Both rows are the harness defaults, like every row here; they measure those defaults, not what these harnesses can do when configured with a shell or a browser.

Abstract

The As published leaderboard passed 73% of 163 intent tasks across 13 cells. The leader is deepseek / claude-haiku-4-5-20251001. Runs achieved this pass rate at a median cost of $0.02 (mean $0.04), with a median token count of 81k (mean 134.3k) and median duration of 26.4s (mean 45.1s). The benchmark's own verdict decided the ranking: pass rate was the deterministic signal that separated the setups.

The result

Color by

aiderclaude-agent-sdkclaude-codecodexcontinuedeepagentsdeepseekevehermesopenclawopencode

Best setup: DeepSeek Harness / Claude Haiku 4.5 (83.0 of 100)

  1. 1DeepSeek Harness / Claude Haiku 4.583.0best
  2. 2eve / Claude Haiku 4.581.5
  3. 3Codex / GPT-5.4 Mini80.0
  4. 4Hermes / Claude Haiku 4.577.9
  5. 5Claude Agent SDK / Claude Haiku 4.574.6
  6. 6Continue / Claude Haiku 4.573.8
  7. 7Codex / GPT-6.1 Sol72.8
  8. 8Claude Code / Claude Sonnet 5.571.3
  9. 9Claude Code / Claude Haiku 4.567.0
  10. 10opencode / Claude Haiku 4.560.5
  11. 11Aider / Claude Haiku 4.521.2
  12. 12OpenClaw / Claude Haiku 4.516.2
  13. 13DeepAgents / Claude Haiku 4.511.8

Overall score per cell, 0–100 points (not a pass rate): 75% benchmark pass rate (the benchmark's own graders carry the outcome share — no judge grades or evals on this board) + 25% efficiency (cost · tokens · duration, vs the board's best agent). Absent components renormalize. Best at the top — the board's best setup is tagged.

Leaderboard

Cells ranked by the composite score — outcome (the lenses that graded) × efficiency — absolute values, no other group. Tokens, cost and duration are per-cell medians, the efficiency the score folds.

#HarnessModelScorePass rateQualityEvalsGoalTokensCostDurationRuns
1DeepSeek HarnessClaude Haiku 4.583.0100%−−−76.6k$0.0211.7s162/163
2eveClaude Haiku 4.581.599%−−−44.8k$0.0213.5s163/163
3CodexGPT-5.4 Mini80.0100%−−−55.3k$0.0226.6s163/163
4HermesClaude Haiku 4.577.9100%−−−122.7k$0.0250.6s163/163
5Claude Agent SDKClaude Haiku 4.574.693%−−−165.4k$0.0316.9s163/163
6ContinueClaude Haiku 4.573.890%−−−15.1k$0.0225.5s162/163
7CodexGPT-6.1 Sol72.896%−−−300.2k$0.091m 41s163/163
8Claude CodeClaude Sonnet 5.571.391%−−−129.3k$0.0528.3s163/163
9Claude CodeClaude Haiku 4.567.085%−−−187.3k$0.0433.4s163/163
10opencodeClaude Haiku 4.560.573%−−−41.2k$0.0128.5s163/163
11AiderClaude Haiku 4.521.22%−−−2.9k$0.003926.4s163/163
12OpenClawClaude Haiku 4.516.218%−−−150.5k$0.0258.7s163/163
13DeepAgentsClaude Haiku 4.511.81%−−−21.1k$0.028.3s163/163

Pass rate

77.5% (95% CI 76.6–78.3)over 1372 graded runs
  • Task text is withheld from public pages until an admin verifies this licence. Running this benchmark is unaffected.
CellPass rateTasksRunsRan asMatcher
aider/claude-haiku-4-5-202510013.8% (95% CI 0.9–7.6)106106 (+57 unmeasured)harness defaultjson_fields
claude-agent-sdk/claude-haiku-4-5-2025100196.2% (95% CI 92.5–99.1)106106 (+57 unmeasured)harness defaultjson_fields
claude-code/claude-haiku-4-5-2025100196.2% (95% CI 92.5–99.1)106106 (+57 unmeasured)harness defaultjson_fields
claude-code/claude-sonnet-5-5100%106106 (+57 unmeasured)harness defaultjson_fields
codex/gpt-5.4-mini100%106106 (+57 unmeasured)harness defaultjson_fields
codex/gpt-6.1-sol99.1% (95% CI 96.2–100)105105 (+58 unmeasured)harness defaultjson_fields
continue/claude-haiku-4-5-2025100197.1% (95% CI 93.3–100)105105 (+58 unmeasured)harness defaultjson_fields
deepagents/claude-haiku-4-5-202510010.9% (95% CI 0–2.8)106106 (+57 unmeasured)harness defaultjson_fields
deepseek/claude-haiku-4-5-20251001100%105105 (+58 unmeasured)harness defaultjson_fields
eve/claude-haiku-4-5-20251001100%106106 (+57 unmeasured)harness defaultjson_fields
hermes/claude-haiku-4-5-20251001100%106106 (+57 unmeasured)harness defaultjson_fields
openclaw/claude-haiku-4-5-2025100121.4% (95% CI 13.6–29.1)103103 (+60 unmeasured)harness defaultjson_fields
opencode/claude-haiku-4-5-2025100191.5% (95% CI 85.9–96.2)106106 (+57 unmeasured)harness defaultjson_fields

Each run's answer was extracted from its response and matched against the benchmark's own key — no model in the loop. Two-stage: the mean over repeats per task, then the mean over tasks, with the interval taken over tasks.

Quality × efficiency clusters — normalized per task

Every graded run, standardized WITHIN its task so difficulty cancels out: → right = fewer tokens than the field on the same task, ↑ up = higher quality index (85% the benchmark's own pass/fail + 15% outcome) than the field. The field pools every harness and model, so a setup's left–right position largely reflects its own token habits. Dots are runs; each harness logo is a harness × model × group centroid — hover it for the model and averages. Up-right wins.

Label
Task
Harness
Model
2091 of 2091 runs match
Show
Color by
better · cheaperworse · pricier
← pricier than the fieldefficiency (σ, per task)cheaper than the field →

28 runs not plotted (missing a grade or token count — infra failures included).

Every metric, per harness × model

Benchmark pass rate

  1. Codex / GPT-5.4 Mini100%
  2. DeepSeek Harness / Claude Haiku 4.5100%
  3. Hermes / Claude Haiku 4.5100%
  4. eve / Claude Haiku 4.599%
  5. Codex / GPT-6.1 Sol96%
  6. Claude Agent SDK / Claude Haiku 4.593%
  7. Claude Code / Claude Sonnet 5.591%
  8. Continue / Claude Haiku 4.590%
  9. Claude Code / Claude Haiku 4.585%
  10. opencode / Claude Haiku 4.573%
  11. OpenClaw / Claude Haiku 4.518%
  12. Aider / Claude Haiku 4.52%
  13. DeepAgents / Claude Haiku 4.51%

Task completion

  1. Aider / Claude Haiku 4.5100%
  2. Claude Agent SDK / Claude Haiku 4.5100%
  3. Claude Code / Claude Haiku 4.5100%
  4. Claude Code / Claude Sonnet 5.5100%
  5. Codex / GPT-5.4 Mini100%
  6. Continue / Claude Haiku 4.5100%
  7. DeepAgents / Claude Haiku 4.5100%
  8. DeepSeek Harness / Claude Haiku 4.5100%
  9. eve / Claude Haiku 4.5100%
  10. Hermes / Claude Haiku 4.5100%
  11. opencode / Claude Haiku 4.5100%
  12. Codex / GPT-6.1 Sol99%
  13. OpenClaw / Claude Haiku 4.597%

Cost per run

  1. Aider / Claude Haiku 4.5$0.0039
  2. opencode / Claude Haiku 4.5$0.01
  3. Codex / GPT-5.4 Mini$0.02
  4. Continue / Claude Haiku 4.5$0.02
  5. DeepAgents / Claude Haiku 4.5$0.02
  6. DeepSeek Harness / Claude Haiku 4.5$0.02
  7. OpenClaw / Claude Haiku 4.5$0.02
  8. eve / Claude Haiku 4.5$0.02
  9. Hermes / Claude Haiku 4.5$0.02
  10. Claude Agent SDK / Claude Haiku 4.5$0.03
  11. Claude Code / Claude Haiku 4.5$0.04
  12. Claude Code / Claude Sonnet 5.5$0.05
  13. Codex / GPT-6.1 Sol$0.09

Tokens per run

  1. Aider / Claude Haiku 4.52.9k
  2. Continue / Claude Haiku 4.515.1k
  3. DeepAgents / Claude Haiku 4.521.1k
  4. opencode / Claude Haiku 4.541.2k
  5. eve / Claude Haiku 4.544.8k
  6. Codex / GPT-5.4 Mini55.3k
  7. DeepSeek Harness / Claude Haiku 4.576.6k
  8. Hermes / Claude Haiku 4.5122.7k
  9. Claude Code / Claude Sonnet 5.5129.3k
  10. OpenClaw / Claude Haiku 4.5150.5k
  11. Claude Agent SDK / Claude Haiku 4.5165.4k
  12. Claude Code / Claude Haiku 4.5187.3k
  13. Codex / GPT-6.1 Sol300.2k

Duration per run

  1. DeepAgents / Claude Haiku 4.58.3s
  2. DeepSeek Harness / Claude Haiku 4.511.7s
  3. eve / Claude Haiku 4.513.5s
  4. Claude Agent SDK / Claude Haiku 4.516.9s
  5. Continue / Claude Haiku 4.525.5s
  6. Aider / Claude Haiku 4.526.4s
  7. Codex / GPT-5.4 Mini26.6s
  8. Claude Code / Claude Sonnet 5.528.3s
  9. opencode / Claude Haiku 4.528.5s
  10. Claude Code / Claude Haiku 4.533.4s
  11. Hermes / Claude Haiku 4.550.6s
  12. OpenClaw / Claude Haiku 4.558.7s
  13. Codex / GPT-6.1 Sol1m 41s

By harness, by model

Benchmark pass rate

By harness

  1. DeepSeek Harness100%
  2. Hermes100%
  3. eve99%
  4. Codex98%
  5. Claude Agent SDK93%
  6. Continue90%
  7. Claude Code87%
  8. opencode73%
  9. OpenClaw18%
  10. Aider2%
  11. DeepAgents1%

By model

  1. GPT-5.4 Mini100%
  2. GPT-6.1 Sol96%
  3. Claude Sonnet 5.591%
  4. Claude Haiku 4.566%

Task completion

By harness

  1. Aider100%
  2. Claude Agent SDK100%
  3. Claude Code100%
  4. Continue100%
  5. DeepAgents100%
  6. DeepSeek Harness100%
  7. eve100%
  8. Hermes100%
  9. opencode100%
  10. Codex99%
  11. OpenClaw97%

By model

  1. Claude Sonnet 5.5100%
  2. GPT-5.4 Mini100%
  3. Claude Haiku 4.5100%
  4. GPT-6.1 Sol99%

Cost per run

By harness

  1. Aider$0.0039
  2. opencode$0.01
  3. Continue$0.02
  4. DeepAgents$0.02
  5. DeepSeek Harness$0.02
  6. OpenClaw$0.02
  7. eve$0.02
  8. Hermes$0.02
  9. Claude Agent SDK$0.03
  10. Claude Code$0.05
  11. Codex$0.05

By model

  1. GPT-5.4 Mini$0.02
  2. Claude Haiku 4.5$0.02
  3. Claude Sonnet 5.5$0.05
  4. GPT-6.1 Sol$0.09

Tokens per run

By harness

  1. Aider2.9k
  2. Continue15.1k
  3. DeepAgents21.1k
  4. opencode41.2k
  5. eve44.8k
  6. DeepSeek Harness76.6k
  7. Hermes122.7k
  8. OpenClaw150.5k
  9. Claude Code161.5k
  10. Claude Agent SDK165.4k
  11. Codex176.1k

By model

  1. GPT-5.4 Mini55.3k
  2. Claude Haiku 4.570.2k
  3. Claude Sonnet 5.5129.3k
  4. GPT-6.1 Sol300.2k

Duration per run

By harness

  1. DeepAgents8.3s
  2. DeepSeek Harness11.7s
  3. eve13.5s
  4. Claude Agent SDK16.9s
  5. Continue25.5s
  6. Aider26.4s
  7. opencode28.5s
  8. Claude Code31.4s
  9. Hermes50.6s
  10. OpenClaw58.7s
  11. Codex1m 05s

By model

  1. Claude Haiku 4.526.3s
  2. GPT-5.4 Mini26.6s
  3. Claude Sonnet 5.528.3s
  4. GPT-6.1 Sol1m 41s

Distributions

Every completed run is one dot — the spread the averages hide. Click a dot to replay that run's journey.

Cost per run

As published
Aider · Claude Haiku 4.5
Claude Agent SDK · Claude Haiku 4.5
Claude Code · Claude Haiku 4.5
Claude Code · Claude Sonnet 5.5
Codex · GPT-5.4 Mini
Codex · GPT-6.1 Sol
Continue · Claude Haiku 4.5
DeepAgents · Claude Haiku 4.5
DeepSeek Harness · Claude Haiku 4.5
eve · Claude Haiku 4.5
Hermes · Claude Haiku 4.5
OpenClaw · Claude Haiku 4.5
opencode · Claude Haiku 4.5
0$0.93

Token composition — average per run

InputOutputCache readCache write
Aider · Claude Haiku 4.5 · As published
2.9k
Claude Agent SDK · Claude Haiku 4.5 · As published
181.8k
Claude Code · Claude Haiku 4.5 · As published
197.4k
Claude Code · Claude Sonnet 5.5 · As published
138.2k
Codex · GPT-5.4 Mini · As published
66.1k
Codex · GPT-6.1 Sol · As published
468.4k
Continue · Claude Haiku 4.5 · As published
45.3k
DeepAgents · Claude Haiku 4.5 · As published
96.7k
DeepSeek Harness · Claude Haiku 4.5 · As published
121.9k
eve · Claude Haiku 4.5 · As published
65.7k
Hermes · Claude Haiku 4.5 · As published
131k
OpenClaw · Claude Haiku 4.5 · As published
178k
opencode · Claude Haiku 4.5 · As published
55.9k

Statistics

MetricGroupnMeanMedian (pooled)MinMaxStd dev
CostAs published2114/2119$0.04$0.02$0.0023$0.93$0.05
DurationAs published2119/211945.3s28.4s4s30m 00s1m 39s
Total tokensAs published2114/2119134.3k90.5k2.5k2.6M190.1k
Output tokensAs published2114/21191.3k8445287.2k3.2k
Cache-read tokensAs published2114/2119120k72.7k02.2M180.4k
Cache-write tokensAs published2114/21194.7k3.7k0187.8k8.3k
TurnsAs published1325/21197.1511206.9

n = runs carrying the fact / completed runs in the group; every statistic runs over present facts only — a missing fact is never counted as 0. Std dev is the sample form (n−1), withheld below n = 2. Every figure here POOLS all runs. The typical-run figures in the abstract, the head-to-head decision and the per-setup panels take each task's median first and then the median across tasks, so a task that ran more often never outweighs one that ran once; the two medians can differ. A promise study's headline (“cut average tokens”) compares per-group means.

What each setup did

Derived from each run's recorded tool calls — not from a model's description of the run — and aggregated per setup, so a behavior seen across several runs is stated once with its rate. Open a finding to see the runs behind it, each linked to its journey at the step where it happened.

Harness
Model
Task
Pick a finding or a setup to list the runs behind it.
Claude Agent SDK · anthropic · Claude Haiku 4.5 · As published5 findings
  • 8 of 163 runs · 8 completed
  • 5 of 163 runs · 5 completed
  • 5 of 163 runs · 5 completed
  • 5 of 163 runs · 5 completed
  • 1 of 163 runs · 1 completed
Claude Code · anthropic · Claude Haiku 4.5 · As published4 findings
  • 20 of 163 runs · 20 completed
  • 2 of 163 runs · 2 completed
  • 2 of 163 runs · 2 completed
  • 2 of 163 runs · 2 completed
Codex · openai · GPT-6.1 Sol · As published6 findings
  • 94 of 163 runs · 94 completed
  • 85 of 163 runs · 85 completed
  • 54 of 163 runs · 54 completed
  • 49 of 163 runs · 49 completed
  • 11 of 163 runs · 11 completed
  • 2 of 163 runs · 0 completed
Continue · anthropic · Claude Haiku 4.5 · As published1 finding
  • 1 of 162 runs · 1 completed
DeepSeek Harness · anthropic · Claude Haiku 4.5 · As published3 findings
  • 1 of 162 runs · 1 completed
  • 1 of 162 runs · 1 completed
  • 1 of 162 runs · 1 completed
eve · anthropic · Claude Haiku 4.5 · As published2 findings
  • 1 of 163 runs · 1 completed
  • 1 of 163 runs · 1 completed
Hermes · anthropic · Claude Haiku 4.5 · As published2 findings
  • 3 of 163 runs · 3 completed
  • 1 of 163 runs · 1 completed
opencode · anthropic · Claude Haiku 4.5 · As published1 finding
  • 21 of 163 runs · 21 completed

Did each setup use its subject?

Read from each run's recorded steps. Runs on a harness whose evidence channel is not proven, and runs whose trace is incomplete, are not measured — never counted as unused.

  • DeepSeek Harness · anthropic · Claude Haiku 4.5 · As published4 of 161 measured runs (2%) · 2 not measured

Task text withheld

Task text withheld — this benchmark is guarded and its tasks are not republished here.

Tasks

Runs

Every run is inspectable — open one to replay the agent's journey step by step, with the analyst's read underneath.

Every run, filterable… of 2119Show runs
Task
Harness
Model
Outcome
Status
… of 2119 runs match
Sort
HarnessModelTaskCompletedPassTokensCostDuration

Loading runs…

Methodology

What each metric means
Completed
The run finished and the harness returned a response. It does NOT mean the answer was correct — a completed run can score zero on quality.
Denominator: Terminal runs, excluding those killed by our own infrastructure.
Quality index
A composite used only in the quality x efficiency plot, built from whichever grader scored this study's runs — the judge's rubric, the pre-registered checks or the benchmark's own verdict — plus whether the run's own outcome was success. The map's caption names the grader and its weights.
Denominator: Graded runs — runs the study's grader scored.
Infra-excluded
A run killed by our own infrastructure. It is a missing measurement, never a loss for the group, and is excluded from every rate denominator.
Denominator: Reported as a count beside every affected panel.
One group
Every task runs once per harness × model cell — a leaderboard with no other group. The readout is absolute (quality, success, tokens, cost) and the cells are ranked into a leaderboard.
Run budget and harness builds
Each run had a wall-clock budget and ran on a recorded harness build — disclosed, never used to split a comparison: Aider: 30 min per run, 1 harness build; Claude Agent SDK: 30 min per run, 1 harness build; Claude Code: 30 min per run, 1 harness build; Codex: 30 min per run, 1 harness build; Continue: 30 min per run, 1 harness build; DeepAgents: 30 min per run, 1 harness build; DeepSeek Harness: 30 min per run, 1 harness build; eve: 30 min per run, 1 harness build; Hermes: 30 min per run, 1 harness build; OpenClaw: 30 min per run, 1 harness build; opencode: 30 min per run, 1 harness build (1 of 163 runs not recorded).
One run per task
Every task ran once per cell, in one phrasing. Run-to-run variation is therefore not measured: a single task's difference can be one lucky or unlucky attempt. Every row of the reference table treats the TASKS as the unit: its intervals resample the tasks and its p-values come from a paired test over them, so they describe how much the result depends on which tasks were drawn — not how it would change if the same tasks were run again. With few tasks no difference can reach significance (five tasks cannot go below p = 1/16).
Sample size
13 cells × 163 tasks × 1 group = 2119 runs.