kalanuStudiesLeaderboard

As published: as published (nothing added)

Measured · Study2views

kalanu index seed — GPQA Diamond — 13 harness setups

GPQA Diamond · 198 tasks · 11 harnesses · 4 models

Codex / gpt-6.1-sol led the shootout on GPQA Diamond.

Ranked leaderboard of agents under As published: cells ranked by the composite score — the benchmark's own graders, then efficiency — codex / gpt-6.1-sol leads.

From the publisher

kalanu.ai index seed (owner-approved T1 tier, 2026-10-01). The full GPQA Diamond corpus run once through each of 13 agent harnesses at their default configuration (twelve harnesses on their default model plus two frontier rows), graded by the benchmark's own reference answers or verifier. One repeat per task. nanoclaw joins in a follow-up study once its runtime fix is live.

Abstract

Runs across 13 setups passed GPQA Diamond's 198 tasks at a rate of 71%, with the leader codex / gpt-6.1-sol setting the benchmark's performance bar. Efficiency varied across the cells: As published came in at a median cost of $0.02 (mean $0.03), median tokens of 17.1k (mean 35k), and median duration of 24.3s (mean 42.9s). The leaderboard reflects performance across harness and model combinations that are not crossed in this matrix, so results cannot be attributed to harness or model alone.

The result

Color by

aiderclaude-agent-sdkclaude-codecodexcontinuedeepagentsdeepseekevehermesopenclawopencode

Best setup: Codex / GPT-6.1 Sol (79.5 of 100)

  1. 1Codex / GPT-6.1 Sol79.5best
  2. 2Claude Code / Claude Sonnet 5.577.4
  3. 3Codex / GPT-5.4 Mini71.1
  4. 4Aider / Claude Haiku 4.565.7
  5. 5Continue / Claude Haiku 4.565.7
  6. 6DeepAgents / Claude Haiku 4.563.3
  7. 7Hermes / Claude Haiku 4.561.5
  8. 8eve / Claude Haiku 4.557.0
  9. 9opencode / Claude Haiku 4.556.9
  10. 10Claude Code / Claude Haiku 4.556.6
  11. 11DeepSeek Harness / Claude Haiku 4.555.3
  12. 12OpenClaw / Claude Haiku 4.554.8
  13. 13Claude Agent SDK / Claude Haiku 4.554.0

Overall score per cell, 0–100 points (not a pass rate): 75% benchmark pass rate (the benchmark's own graders carry the outcome share — no judge grades or evals on this board) + 25% efficiency (cost · tokens · duration, vs the board's best agent). Absent components renormalize. Best at the top — the board's best setup is tagged.

Leaderboard

Cells ranked by the composite score — outcome (the lenses that graded) × efficiency — absolute values, no other group. Tokens, cost and duration are per-cell medians, the efficiency the score folds.

#HarnessModelScorePass rateQualityEvalsGoalTokensCostDurationRuns
1CodexGPT-6.1 Sol79.596%−−−16.1k$0.0320.3s198/198
2Claude CodeClaude Sonnet 5.577.492%−−−29.7k$0.0218.3s198/198
3CodexGPT-5.4 Mini71.179%−−−11.3k$0.0118.3s198/198
4ContinueClaude Haiku 4.565.765%−−−3.4k$0.0122.3s196/198
5AiderClaude Haiku 4.565.761%−−−3.1k$0.0122.3s198/198
6DeepAgentsClaude Haiku 4.563.367%−−−10.6k$0.0211.6s198/198
7HermesClaude Haiku 4.561.567%−−−13.9k$0.0132.6s198/198
8eveClaude Haiku 4.557.065%−−−18.1k$0.0219.4s198/198
9opencodeClaude Haiku 4.556.955%−−−8.8k$0.004722.3s198/198
10Claude CodeClaude Haiku 4.556.671%−−−39.9k$0.0355.6s198/198
11DeepSeek HarnessClaude Haiku 4.555.367%−−−54.9k$0.0331.9s198/198
12OpenClawClaude Haiku 4.554.868%−−−40.7k$0.021m 11s198/198
13Claude Agent SDKClaude Haiku 4.554.067%−−−36.5k$0.0447.8s198/198

Pass rate

70.7% (95% CI 66.5–74.8)over 2568 graded runs
CellPass rateTasksRunsRan asMatcher
aider/claude-haiku-4-5-2025100160.6% (95% CI 54–67.2)198198harness defaultchoice_letter
claude-agent-sdk/claude-haiku-4-5-2025100167% (95% CI 60.4–73.1)197197 (+1 unmeasured)harness defaultchoice_letter
claude-code/claude-haiku-4-5-2025100170.7% (95% CI 64.7–76.8)198198harness defaultchoice_letter
claude-code/claude-sonnet-5-592.4% (95% CI 88.4–96)198198harness defaultchoice_letter
codex/gpt-5.4-mini78.8% (95% CI 73.2–84.3)198198harness defaultchoice_letter
codex/gpt-6.1-sol96% (95% CI 92.9–98.5)198198harness defaultchoice_letter
continue/claude-haiku-4-5-2025100164.6% (95% CI 58–70.8)195195 (+3 unmeasured)harness defaultchoice_letter
deepagents/claude-haiku-4-5-2025100166.8% (95% CI 60.2–73.5)196196 (+2 unmeasured)harness defaultchoice_letter
deepseek/claude-haiku-4-5-2025100167.2% (95% CI 60.6–73.7)198198harness defaultchoice_letter
eve/claude-haiku-4-5-2025100165.2% (95% CI 59.1–72.2)198198harness defaultchoice_letter
hermes/claude-haiku-4-5-2025100166.7% (95% CI 60.1–73.2)198198harness defaultchoice_letter
openclaw/claude-haiku-4-5-2025100168.2% (95% CI 61.6–74.8)198198harness defaultchoice_letter
opencode/claude-haiku-4-5-2025100155.1% (95% CI 48–61.6)198198harness defaultchoice_letter

Each run's answer was extracted from its response and matched against the benchmark's own key — no model in the loop. Two-stage: the mean over repeats per task, then the mean over tasks, with the interval taken over tasks.

Quality × efficiency clusters — normalized per task

Every graded run, standardized WITHIN its task so difficulty cancels out: → right = fewer tokens than the field on the same task, ↑ up = higher quality index (85% the benchmark's own pass/fail + 15% outcome) than the field. The field pools every harness and model, so a setup's left–right position largely reflects its own token habits. Dots are runs; each harness logo is a harness × model × group centroid — hover it for the model and averages. Up-right wins.

Label
Task
Harness
Model
2566 of 2566 runs match
Show
Color by
better · cheaperworse · pricier
← pricier than the fieldefficiency (σ, per task)cheaper than the field →

8 runs not plotted (missing a grade or token count — infra failures included).

Every metric, per harness × model

Benchmark pass rate

  1. Codex / GPT-6.1 Sol96%
  2. Claude Code / Claude Sonnet 5.592%
  3. Codex / GPT-5.4 Mini79%
  4. Claude Code / Claude Haiku 4.571%
  5. OpenClaw / Claude Haiku 4.568%
  6. DeepSeek Harness / Claude Haiku 4.567%
  7. Claude Agent SDK / Claude Haiku 4.567%
  8. DeepAgents / Claude Haiku 4.567%
  9. Hermes / Claude Haiku 4.567%
  10. eve / Claude Haiku 4.565%
  11. Continue / Claude Haiku 4.565%
  12. Aider / Claude Haiku 4.561%
  13. opencode / Claude Haiku 4.555%

Task completion

  1. Aider / Claude Haiku 4.5100%
  2. Claude Code / Claude Haiku 4.5100%
  3. Claude Code / Claude Sonnet 5.5100%
  4. Codex / GPT-5.4 Mini100%
  5. Codex / GPT-6.1 Sol100%
  6. Continue / Claude Haiku 4.5100%
  7. DeepSeek Harness / Claude Haiku 4.5100%
  8. eve / Claude Haiku 4.5100%
  9. Hermes / Claude Haiku 4.5100%
  10. OpenClaw / Claude Haiku 4.5100%
  11. opencode / Claude Haiku 4.5100%
  12. Claude Agent SDK / Claude Haiku 4.599%
  13. DeepAgents / Claude Haiku 4.599%

Cost per run

  1. opencode / Claude Haiku 4.5$0.0047
  2. Aider / Claude Haiku 4.5$0.01
  3. Hermes / Claude Haiku 4.5$0.01
  4. Continue / Claude Haiku 4.5$0.01
  5. Codex / GPT-5.4 Mini$0.01
  6. DeepAgents / Claude Haiku 4.5$0.02
  7. Claude Code / Claude Sonnet 5.5$0.02
  8. eve / Claude Haiku 4.5$0.02
  9. OpenClaw / Claude Haiku 4.5$0.02
  10. DeepSeek Harness / Claude Haiku 4.5$0.03
  11. Codex / GPT-6.1 Sol$0.03
  12. Claude Code / Claude Haiku 4.5$0.03
  13. Claude Agent SDK / Claude Haiku 4.5$0.04

Tokens per run

  1. Aider / Claude Haiku 4.53.1k
  2. Continue / Claude Haiku 4.53.4k
  3. opencode / Claude Haiku 4.58.8k
  4. DeepAgents / Claude Haiku 4.510.6k
  5. Codex / GPT-5.4 Mini11.3k
  6. Hermes / Claude Haiku 4.513.9k
  7. Codex / GPT-6.1 Sol16.1k
  8. eve / Claude Haiku 4.518.1k
  9. Claude Code / Claude Sonnet 5.529.7k
  10. Claude Agent SDK / Claude Haiku 4.536.5k
  11. Claude Code / Claude Haiku 4.539.9k
  12. OpenClaw / Claude Haiku 4.540.7k
  13. DeepSeek Harness / Claude Haiku 4.554.9k

Duration per run

  1. DeepAgents / Claude Haiku 4.511.6s
  2. Claude Code / Claude Sonnet 5.518.3s
  3. Codex / GPT-5.4 Mini18.3s
  4. eve / Claude Haiku 4.519.4s
  5. Codex / GPT-6.1 Sol20.3s
  6. opencode / Claude Haiku 4.522.3s
  7. Continue / Claude Haiku 4.522.3s
  8. Aider / Claude Haiku 4.522.3s
  9. DeepSeek Harness / Claude Haiku 4.531.9s
  10. Hermes / Claude Haiku 4.532.6s
  11. Claude Agent SDK / Claude Haiku 4.547.8s
  12. Claude Code / Claude Haiku 4.555.6s
  13. OpenClaw / Claude Haiku 4.51m 11s

By harness, by model

Benchmark pass rate

By harness

  1. Codex87%
  2. Claude Code82%
  3. OpenClaw68%
  4. DeepSeek Harness67%
  5. Claude Agent SDK67%
  6. DeepAgents67%
  7. Hermes67%
  8. eve65%
  9. Continue65%
  10. Aider61%
  11. opencode55%

By model

  1. GPT-6.1 Sol96%
  2. Claude Sonnet 5.592%
  3. GPT-5.4 Mini79%
  4. Claude Haiku 4.565%

Task completion

By harness

  1. Aider100%
  2. Claude Code100%
  3. Codex100%
  4. Continue100%
  5. DeepSeek Harness100%
  6. eve100%
  7. Hermes100%
  8. OpenClaw100%
  9. opencode100%
  10. Claude Agent SDK99%
  11. DeepAgents99%

By model

  1. Claude Sonnet 5.5100%
  2. GPT-5.4 Mini100%
  3. GPT-6.1 Sol100%
  4. Claude Haiku 4.5100%

Cost per run

By harness

  1. opencode$0.0047
  2. Aider$0.01
  3. Hermes$0.01
  4. Continue$0.01
  5. DeepAgents$0.02
  6. Codex$0.02
  7. eve$0.02
  8. OpenClaw$0.02
  9. Claude Code$0.03
  10. DeepSeek Harness$0.03
  11. Claude Agent SDK$0.04

By model

  1. GPT-5.4 Mini$0.01
  2. Claude Haiku 4.5$0.02
  3. Claude Sonnet 5.5$0.02
  4. GPT-6.1 Sol$0.03

Tokens per run

By harness

  1. Aider3.1k
  2. Continue3.4k
  3. opencode8.8k
  4. DeepAgents10.6k
  5. Codex13.6k
  6. Hermes13.9k
  7. eve18.1k
  8. Claude Code34.9k
  9. Claude Agent SDK36.5k
  10. OpenClaw40.7k
  11. DeepSeek Harness54.9k

By model

  1. GPT-5.4 Mini11.3k
  2. GPT-6.1 Sol16.1k
  3. Claude Haiku 4.522.3k
  4. Claude Sonnet 5.529.7k

Duration per run

By harness

  1. DeepAgents11.6s
  2. eve19.4s
  3. Codex20.3s
  4. opencode22.3s
  5. Continue22.3s
  6. Aider22.3s
  7. DeepSeek Harness31.9s
  8. Hermes32.6s
  9. Claude Code38.5s
  10. Claude Agent SDK47.8s
  11. OpenClaw1m 11s

By model

  1. Claude Sonnet 5.518.3s
  2. GPT-5.4 Mini18.3s
  3. GPT-6.1 Sol20.3s
  4. Claude Haiku 4.527.5s

Distributions

Every completed run is one dot — the spread the averages hide. Click a dot to replay that run's journey.

Cost per run

As published
Aider · Claude Haiku 4.5
Claude Agent SDK · Claude Haiku 4.5
Claude Code · Claude Haiku 4.5
Claude Code · Claude Sonnet 5.5
Codex · GPT-5.4 Mini
Codex · GPT-6.1 Sol
Continue · Claude Haiku 4.5
DeepAgents · Claude Haiku 4.5
DeepSeek Harness · Claude Haiku 4.5
eve · Claude Haiku 4.5
Hermes · Claude Haiku 4.5
OpenClaw · Claude Haiku 4.5
opencode · Claude Haiku 4.5
0$0.52

Token composition — average per run

InputOutputCache readCache write
Aider · Claude Haiku 4.5 · As published
3.5k
Claude Agent SDK · Claude Haiku 4.5 · As published
40.2k
Claude Code · Claude Haiku 4.5 · As published
45.2k
Claude Code · Claude Sonnet 5.5 · As published
31.4k
Codex · GPT-5.4 Mini · As published
16.7k
Codex · GPT-6.1 Sol · As published
16.7k
Continue · Claude Haiku 4.5 · As published
12.5k
DeepAgents · Claude Haiku 4.5 · As published
69.6k
DeepSeek Harness · Claude Haiku 4.5 · As published
92.6k
eve · Claude Haiku 4.5 · As published
36.9k
Hermes · Claude Haiku 4.5 · As published
26.9k
OpenClaw · Claude Haiku 4.5 · As published
48.2k
opencode · Claude Haiku 4.5 · As published
14.1k

Statistics

MetricGroupnMeanMedian (pooled)MinMaxStd dev
CostAs published2566/2574$0.03$0.02$0.0013$0.52$0.04
DurationAs published2574/257442.9s24.3s3.5s16m 03s51.7s
Total tokensAs published2566/257435k18.6k2.5k1.3M71.5k
Output tokensAs published2566/25743.3k1.1k2454.6k5.3k
Cache-read tokensAs published2566/257424.9k12.5k01.2M62.7k
Cache-write tokensAs published2566/25742.8k6210103.3k7k
TurnsAs published1616/25742.621503.7

n = runs carrying the fact / completed runs in the group; every statistic runs over present facts only — a missing fact is never counted as 0. Std dev is the sample form (n−1), withheld below n = 2. Every figure here POOLS all runs. The typical-run figures in the abstract, the head-to-head decision and the per-setup panels take each task's median first and then the median across tasks, so a task that ran more often never outweighs one that ran once; the two medians can differ. A promise study's headline (“cut average tokens”) compares per-group means.

What each setup did

Derived from each run's recorded tool calls — not from a model's description of the run — and aggregated per setup, so a behavior seen across several runs is stated once with its rate. Open a finding to see the runs behind it, each linked to its journey at the step where it happened.

Harness
Model
Task
Pick a finding or a setup to list the runs behind it.
Aider · anthropic · Claude Haiku 4.5 · As published1 finding
  • 188 of 198 runs · 188 completed
Claude Agent SDK · anthropic · Claude Haiku 4.5 · As published3 findings
  • 194 of 198 runs · 193 completed
  • 2 of 198 runs · 2 completed
  • 2 of 198 runs · 2 completed
Claude Code · anthropic · Claude Haiku 4.5 · As published3 findings
  • 195 of 198 runs · 195 completed
  • 2 of 198 runs · 2 completed
  • 1 of 198 runs · 1 completed
Claude Code · anthropic · Claude Sonnet 5.5 · As published4 findings
  • 195 of 198 runs · 195 completed
  • 3 of 198 runs · 3 completed
  • 1 of 198 runs · 1 completed
  • 1 of 198 runs · 1 completed
Codex · openai · GPT-5.4 Mini · As published1 finding
  • 181 of 198 runs · 181 completed
Codex · openai · GPT-6.1 Sol · As published2 findings
  • 82 of 198 runs · 82 completed
  • 1 of 198 runs · 1 completed
Continue · anthropic · Claude Haiku 4.5 · As published3 findings
  • 118 of 196 runs · 118 completed
  • 11 of 196 runs · 11 completed
  • 3 of 196 runs · 3 completed
DeepAgents · anthropic · Claude Haiku 4.5 · As published2 findings
  • 95 of 198 runs · 95 completed
  • 4 of 198 runs · 3 completed
DeepSeek Harness · anthropic · Claude Haiku 4.5 · As published3 findings
  • 38 of 198 runs · 38 completed
  • 30 of 198 runs · 30 completed
  • 7 of 198 runs · 7 completed
eve · anthropic · Claude Haiku 4.5 · As published3 findings
  • 48 of 198 runs · 48 completed
  • 18 of 198 runs · 18 completed
  • 5 of 198 runs · 5 completed
Hermes · anthropic · Claude Haiku 4.5 · As published2 findings
  • 158 of 198 runs · 158 completed
  • 10 of 198 runs · 10 completed
OpenClaw · anthropic · Claude Haiku 4.5 · As published3 findings
  • 193 of 198 runs · 193 completed
  • 1 of 198 runs · 1 completed
  • 1 of 198 runs · 1 completed
opencode · anthropic · Claude Haiku 4.5 · As published2 findings
  • 180 of 198 runs · 180 completed
  • 10 of 198 runs · 10 completed

Did each setup use its subject?

Read from each run's recorded steps. Runs on a harness whose evidence channel is not proven, and runs whose trace is incomplete, are not measured — never counted as unused.

  • DeepSeek Harness · anthropic · Claude Haiku 4.5 · As published19 of 195 measured runs (10%) · 3 not measured

Task text withheld

Task text withheld — this benchmark is guarded and its tasks are not republished here.

Tasks

Runs

Every run is inspectable — open one to replay the agent's journey step by step, with the analyst's read underneath.

Every run, filterable… of 2574Show runs
Task
Harness
Model
Outcome
Status
… of 2574 runs match
Sort
HarnessModelTaskCompletedPassTokensCostDuration

Loading runs…

Methodology

What each metric means
Completed
The run finished and the harness returned a response. It does NOT mean the answer was correct — a completed run can score zero on quality.
Denominator: Terminal runs, excluding those killed by our own infrastructure.
Quality index
A composite used only in the quality x efficiency plot, built from whichever grader scored this study's runs — the judge's rubric, the pre-registered checks or the benchmark's own verdict — plus whether the run's own outcome was success. The map's caption names the grader and its weights.
Denominator: Graded runs — runs the study's grader scored.
Infra-excluded
A run killed by our own infrastructure. It is a missing measurement, never a loss for the group, and is excluded from every rate denominator.
Denominator: Reported as a count beside every affected panel.
One group
Every task runs once per harness × model cell — a leaderboard with no other group. The readout is absolute (quality, success, tokens, cost) and the cells are ranked into a leaderboard.
Run budget and harness builds
Each run had a wall-clock budget and ran on a recorded harness build — disclosed, never used to split a comparison: Aider: 8 min per run, 1 harness build; Claude Agent SDK: 8 min per run, 1 harness build; Claude Code: 30 min per run, 1 harness build; Codex: 30 min per run, 1 harness build; Continue: 8 min per run, 1 harness build; DeepAgents: 8 min per run, 1 harness build; DeepSeek Harness: 30 min per run, 1 harness build; eve: 8 min per run, 1 harness build; Hermes: 8 min per run, 1 harness build; OpenClaw: 30 min per run, 1 harness build; opencode: 30 min per run, 1 harness build (1 of 198 runs not recorded).
One run per task
Every task ran once per cell, in one phrasing. Run-to-run variation is therefore not measured: a single task's difference can be one lucky or unlucky attempt. Every row of the reference table treats the TASKS as the unit: its intervals resample the tasks and its p-values come from a paired test over them, so they describe how much the result depends on which tasks were drawn — not how it would change if the same tasks were run again. With few tasks no difference can reach significance (five tasks cannot go below p = 1/16).
Sample size
13 cells × 198 tasks × 1 group = 2574 runs.