kalanuStudiesLeaderboard

As published: as published (nothing added)

Measured · Study1views

kalanu index seed — IFEval — 13 harness setups

IFEval · 69 tasks · 11 harnesses · 4 models

Codex / gpt-5.4-mini leads the IFEval leaderboard.

Ranked leaderboard of agents under As published: cells ranked by the composite score — the pre-registered checks, then efficiency — codex / gpt-5.4-mini leads.

From the publisher

kalanu.ai index seed (owner-approved D4, 2026-10-03). The full IFEval corpus (69 tasks) run once through each of 13 agent harness setups at their default configuration (eleven harnesses on their default model plus two frontier rows: claude-code on claude-sonnet-5-5 and codex on gpt-6.1-sol), graded by the benchmark's own grader. One repeat per task. nanoclaw joins in a follow-up study once its runtime fix is live.

Abstract

Across 69 tasks over 13 harness/model cells in a leaderboard of agents under As published, runs cost a median of $0.01 (mean $0.01), used a median of 11.9k (mean 19.3k) tokens and took a median of 18.3s (mean 21.3s) of wall-clock each. The cells passed 84% of the pre-registered checks. Harness and model are NOT crossed in this matrix (13 cells over 11 harnesses and 4 models) — by-harness and by-model splits share cells, so never attribute an effect to one of them alone.

The result

Color by

aiderclaude-agent-sdkclaude-codecodexcontinuedeepagentsdeepseekevehermesopenclawopencode

Best setup: Codex / GPT-5.4 Mini (86.8 of 100)

  1. 1Codex / GPT-5.4 Mini86.8best
  2. 2Claude Agent SDK / Claude Haiku 4.584.0
  3. 3Claude Code / Claude Sonnet 5.579.1
  4. 4Codex / GPT-6.1 Sol78.8
  5. 5OpenClaw / Claude Haiku 4.577.9
  6. 6Claude Code / Claude Haiku 4.576.1
  7. 7DeepSeek Harness / Claude Haiku 4.571.9
  8. 8DeepAgents / Claude Haiku 4.570.9
  9. 9Aider / Claude Haiku 4.569.1
  10. 10Hermes / Claude Haiku 4.568.1
  11. 11eve / Claude Haiku 4.567.7
  12. 12opencode / Claude Haiku 4.562.8
  13. 13Continue / Claude Haiku 4.547.8

Overall score per cell, 0–100 points (not a pass rate): 75% evals (the deterministic checks carry the outcome share — no judge grades on this board) + 25% efficiency (cost · tokens · duration, vs the board's best agent). Absent components renormalize. Best at the top — the board's best setup is tagged.

Leaderboard

Cells ranked by the composite score — outcome (the lenses that graded) × efficiency — absolute values, no other group. Tokens, cost and duration are per-cell medians, the efficiency the score folds.

#HarnessModelScoreQualityEvalsGoalTokensCostDurationRuns
1CodexGPT-5.4 Mini86.8−100%−10.7k$0.002914.5s69/69
2Claude Agent SDKClaude Haiku 4.584.0−99%−29.9k$0.017.1s69/69
3Claude CodeClaude Sonnet 5.579.1−100%−29.7k$0.0220.6s69/69
4CodexGPT-6.1 Sol78.8−99%−12.1k$0.0322.3s69/69
5OpenClawClaude Haiku 4.577.9−99%−37.4k$0.0148.7s69/69
6Claude CodeClaude Haiku 4.576.1−94%−34.7k$0.0118.3s69/69
7DeepSeek HarnessClaude Haiku 4.571.9−83%−22.3k$0.018.9s69/69
8DeepAgentsClaude Haiku 4.570.9−78%−9k$0.016.4s69/69
9AiderClaude Haiku 4.569.1−71%−2.7k$0.003620.3s69/69
10HermesClaude Haiku 4.568.1−80%−13.3k$0.003630.4s69/69
11eveClaude Haiku 4.567.7−75%−8.5k$0.017.7s69/69
12opencodeClaude Haiku 4.562.8−65%−8.2k$0.002118.3s69/69
13ContinueClaude Haiku 4.547.8−50%−4.7k$0.0118.3s69/69

Quality × efficiency clusters — normalized per task

Every graded run, standardized WITHIN its task so difficulty cancels out: → right = fewer tokens than the field on the same task, ↑ up = higher quality index (85% deterministic checks passed + 15% outcome) than the field. The field pools every harness and model, so a setup's left–right position largely reflects its own token habits. Dots are runs; each harness logo is a harness × model × group centroid — hover it for the model and averages. Up-right wins.

Task
Harness
Model
Evals
893 of 893 runs match
Show
Color by
better · cheaperworse · pricier
← pricier than the fieldefficiency (σ, per task)cheaper than the field →

4 runs not plotted (missing a grade or token count — infra failures included).

Every metric, per harness × model

Task completion

  1. Aider / Claude Haiku 4.5100%
  2. Claude Agent SDK / Claude Haiku 4.5100%
  3. Claude Code / Claude Haiku 4.5100%
  4. Codex / GPT-5.4 Mini100%
  5. Codex / GPT-6.1 Sol100%
  6. Continue / Claude Haiku 4.5100%
  7. DeepAgents / Claude Haiku 4.5100%
  8. DeepSeek Harness / Claude Haiku 4.5100%
  9. eve / Claude Haiku 4.5100%
  10. Hermes / Claude Haiku 4.5100%
  11. OpenClaw / Claude Haiku 4.5100%
  12. opencode / Claude Haiku 4.5100%
  13. Claude Code / Claude Sonnet 5.599%

Evals pass rate

  1. Claude Code / Claude Sonnet 5.5100%
  2. Codex / GPT-5.4 Mini100%
  3. Claude Agent SDK / Claude Haiku 4.599%
  4. Codex / GPT-6.1 Sol99%
  5. OpenClaw / Claude Haiku 4.599%
  6. Claude Code / Claude Haiku 4.594%
  7. DeepSeek Harness / Claude Haiku 4.583%
  8. Hermes / Claude Haiku 4.580%
  9. DeepAgents / Claude Haiku 4.578%
  10. eve / Claude Haiku 4.575%
  11. Aider / Claude Haiku 4.571%
  12. opencode / Claude Haiku 4.565%
  13. Continue / Claude Haiku 4.550%

Cost per run

  1. opencode / Claude Haiku 4.5$0.0021
  2. Codex / GPT-5.4 Mini$0.0029
  3. Aider / Claude Haiku 4.5$0.0036
  4. Hermes / Claude Haiku 4.5$0.0036
  5. DeepSeek Harness / Claude Haiku 4.5$0.01
  6. Continue / Claude Haiku 4.5$0.01
  7. OpenClaw / Claude Haiku 4.5$0.01
  8. Claude Agent SDK / Claude Haiku 4.5$0.01
  9. Claude Code / Claude Haiku 4.5$0.01
  10. eve / Claude Haiku 4.5$0.01
  11. DeepAgents / Claude Haiku 4.5$0.01
  12. Claude Code / Claude Sonnet 5.5$0.02
  13. Codex / GPT-6.1 Sol$0.03

Tokens per run

  1. Aider / Claude Haiku 4.52.7k
  2. Continue / Claude Haiku 4.54.7k
  3. opencode / Claude Haiku 4.58.2k
  4. eve / Claude Haiku 4.58.5k
  5. DeepAgents / Claude Haiku 4.59k
  6. Codex / GPT-5.4 Mini10.7k
  7. Codex / GPT-6.1 Sol12.1k
  8. Hermes / Claude Haiku 4.513.3k
  9. DeepSeek Harness / Claude Haiku 4.522.3k
  10. Claude Code / Claude Sonnet 5.529.7k
  11. Claude Agent SDK / Claude Haiku 4.529.9k
  12. Claude Code / Claude Haiku 4.534.7k
  13. OpenClaw / Claude Haiku 4.537.4k

Duration per run

  1. DeepAgents / Claude Haiku 4.56.4s
  2. Claude Agent SDK / Claude Haiku 4.57.1s
  3. eve / Claude Haiku 4.57.7s
  4. DeepSeek Harness / Claude Haiku 4.58.9s
  5. Codex / GPT-5.4 Mini14.5s
  6. Claude Code / Claude Haiku 4.518.3s
  7. Continue / Claude Haiku 4.518.3s
  8. opencode / Claude Haiku 4.518.3s
  9. Aider / Claude Haiku 4.520.3s
  10. Claude Code / Claude Sonnet 5.520.6s
  11. Codex / GPT-6.1 Sol22.3s
  12. Hermes / Claude Haiku 4.530.4s
  13. OpenClaw / Claude Haiku 4.548.7s

By harness, by model

Task completion

By harness

  1. Aider100%
  2. Claude Agent SDK100%
  3. Codex100%
  4. Continue100%
  5. DeepAgents100%
  6. DeepSeek Harness100%
  7. eve100%
  8. Hermes100%
  9. OpenClaw100%
  10. opencode100%
  11. Claude Code99%

By model

  1. Claude Haiku 4.5100%
  2. GPT-5.4 Mini100%
  3. GPT-6.1 Sol100%
  4. Claude Sonnet 5.599%

Evals pass rate

By harness

  1. Codex99%
  2. Claude Agent SDK99%
  3. OpenClaw99%
  4. Claude Code97%
  5. DeepSeek Harness83%
  6. Hermes80%
  7. DeepAgents78%
  8. eve75%
  9. Aider71%
  10. opencode65%
  11. Continue50%

By model

  1. Claude Sonnet 5.5100%
  2. GPT-5.4 Mini100%
  3. GPT-6.1 Sol99%
  4. Claude Haiku 4.579%

Cost per run

By harness

  1. opencode$0.0021
  2. Aider$0.0036
  3. Hermes$0.0036
  4. DeepSeek Harness$0.01
  5. Continue$0.01
  6. OpenClaw$0.01
  7. Claude Agent SDK$0.01
  8. eve$0.01
  9. DeepAgents$0.01
  10. Claude Code$0.02
  11. Codex$0.02

By model

  1. GPT-5.4 Mini$0.0029
  2. Claude Haiku 4.5$0.01
  3. Claude Sonnet 5.5$0.02
  4. GPT-6.1 Sol$0.03

Tokens per run

By harness

  1. Aider2.7k
  2. Continue4.7k
  3. opencode8.2k
  4. eve8.5k
  5. DeepAgents9k
  6. Codex11.4k
  7. Hermes13.3k
  8. DeepSeek Harness22.3k
  9. Claude Agent SDK29.9k
  10. Claude Code32.2k
  11. OpenClaw37.4k

By model

  1. GPT-5.4 Mini10.7k
  2. Claude Haiku 4.511.2k
  3. GPT-6.1 Sol12.1k
  4. Claude Sonnet 5.529.7k

Duration per run

By harness

  1. DeepAgents6.4s
  2. Claude Agent SDK7.1s
  3. eve7.7s
  4. DeepSeek Harness8.9s
  5. Continue18.3s
  6. opencode18.3s
  7. Codex19.3s
  8. Claude Code19.5s
  9. Aider20.3s
  10. Hermes30.4s
  11. OpenClaw48.7s

By model

  1. GPT-5.4 Mini14.5s
  2. Claude Haiku 4.517.4s
  3. Claude Sonnet 5.520.6s
  4. GPT-6.1 Sol22.3s

Distributions

Every completed run is one dot — the spread the averages hide. Click a dot to replay that run's journey.

Cost per run

As published
Aider · Claude Haiku 4.5
Claude Agent SDK · Claude Haiku 4.5
Claude Code · Claude Haiku 4.5
Claude Code · Claude Sonnet 5.5
Codex · GPT-5.4 Mini
Codex · GPT-6.1 Sol
Continue · Claude Haiku 4.5
DeepAgents · Claude Haiku 4.5
DeepSeek Harness · Claude Haiku 4.5
eve · Claude Haiku 4.5
Hermes · Claude Haiku 4.5
OpenClaw · Claude Haiku 4.5
opencode · Claude Haiku 4.5
0$0.21

Token composition — average per run

InputOutputCache readCache write
Aider · Claude Haiku 4.5 · As published
2.8k
Claude Agent SDK · Claude Haiku 4.5 · As published
34.5k
Claude Code · Claude Haiku 4.5 · As published
37.8k
Claude Code · Claude Sonnet 5.5 · As published
31.6k
Codex · GPT-5.4 Mini · As published
14k
Codex · GPT-6.1 Sol · As published
14.9k
Continue · Claude Haiku 4.5 · As published
5.1k
DeepAgents · Claude Haiku 4.5 · As published
8.4k
DeepSeek Harness · Claude Haiku 4.5 · As published
21.5k
eve · Claude Haiku 4.5 · As published
11.4k
Hermes · Claude Haiku 4.5 · As published
17.2k
OpenClaw · Claude Haiku 4.5 · As published
41.7k
opencode · Claude Haiku 4.5 · As published
9.1k

Statistics

MetricGroupnMeanMedian (pooled)MinMaxStd dev
CostAs published894/897$0.01$0.01$0.0012$0.21$0.01
DurationAs published897/89721.3s18.3s716ms2m 24s15s
Total tokensAs published894/89719.3k12.2k2.2k159.2k16.8k
Output tokensAs published894/897726446016k1.2k
Cache-read tokensAs published894/89714.3k10.1k0148.7k16.7k
Cache-write tokensAs published894/8971.4k366022.7k2.1k
TurnsAs published553/8971.51170.7

n = runs carrying the fact / completed runs in the group; every statistic runs over present facts only — a missing fact is never counted as 0. Std dev is the sample form (n−1), withheld below n = 2. Every figure here POOLS all runs. The typical-run figures in the abstract, the head-to-head decision and the per-setup panels take each task's median first and then the median across tasks, so a task that ran more often never outweighs one that ran once; the two medians can differ. A promise study's headline (“cut average tokens”) compares per-group means.

What each setup did

Derived from each run's recorded tool calls — not from a model's description of the run — and aggregated per setup, so a behavior seen across several runs is stated once with its rate. Open a finding to see the runs behind it, each linked to its journey at the step where it happened.

Harness
Model
Task
Pick a finding or a setup to list the runs behind it.
Aider · anthropic · Claude Haiku 4.5 · As published1 finding
  • 45 of 69 runs · 45 completed
Claude Agent SDK · anthropic · Claude Haiku 4.5 · As published1 finding
  • 61 of 69 runs · 61 completed
Claude Code · anthropic · Claude Haiku 4.5 · As published1 finding
  • 64 of 69 runs · 64 completed
Claude Code · anthropic · Claude Sonnet 5.5 · As published1 finding
  • 67 of 69 runs · 66 completed
Codex · openai · GPT-5.4 Mini · As published1 finding
  • 63 of 69 runs · 63 completed
Codex · openai · GPT-6.1 Sol · As published1 finding
  • 56 of 69 runs · 56 completed
Continue · anthropic · Claude Haiku 4.5 · As published1 finding
  • 35 of 69 runs · 35 completed
DeepAgents · anthropic · Claude Haiku 4.5 · As published1 finding
  • 26 of 69 runs · 26 completed
DeepSeek Harness · anthropic · Claude Haiku 4.5 · As published1 finding
  • 29 of 69 runs · 29 completed
eve · anthropic · Claude Haiku 4.5 · As published1 finding
  • 25 of 69 runs · 25 completed
Hermes · anthropic · Claude Haiku 4.5 · As published1 finding
  • 51 of 69 runs · 51 completed
OpenClaw · anthropic · Claude Haiku 4.5 · As published1 finding
  • 64 of 69 runs · 64 completed
opencode · anthropic · Claude Haiku 4.5 · As published1 finding
  • 63 of 69 runs · 63 completed

Did each setup use its subject?

Read from each run's recorded steps. Runs on a harness whose evidence channel is not proven, and runs whose trace is incomplete, are not measured — never counted as unused.

  • DeepSeek Harness · anthropic · Claude Haiku 4.5 · As published5 of 69 measured runs (7%)

Task text withheld

Task text withheld — this benchmark is guarded and its tasks are not republished here.

Tasks

Runs

Every run is inspectable — open one to replay the agent's journey step by step, with the analyst's read underneath.

Every run, filterable… of 897Show runs
Task
Harness
Model
Outcome
Status
… of 897 runs match
Sort
HarnessModelTaskCompletedQualityEvalsTokensCostDuration

Loading runs…

Methodology

What each metric means
Completed
The run finished and the harness returned a response. It does NOT mean the answer was correct — a completed run can score zero on quality.
Denominator: Terminal runs, excluding those killed by our own infrastructure.
Quality index
A composite used only in the quality x efficiency plot, built from whichever grader scored this study's runs — the judge's rubric, the pre-registered checks or the benchmark's own verdict — plus whether the run's own outcome was success. The map's caption names the grader and its weights.
Denominator: Graded runs — runs the study's grader scored.
Infra-excluded
A run killed by our own infrastructure. It is a missing measurement, never a loss for the group, and is excluded from every rate denominator.
Denominator: Reported as a count beside every affected panel.
One group
Every task runs once per harness × model cell — a leaderboard with no other group. The readout is absolute (quality, success, tokens, cost) and the cells are ranked into a leaderboard.
Run budget and harness builds
Each run had a wall-clock budget and ran on a recorded harness build — disclosed, never used to split a comparison: Aider: 8 min per run, 1 harness build; Claude Agent SDK: 8 min per run, 1 harness build; Claude Code: 30 min per run, 1 harness build; Codex: 30 min per run, 1 harness build; Continue: 8 min per run, 1 harness build; DeepAgents: 8 min per run, 1 harness build; DeepSeek Harness: 30 min per run, 1 harness build; eve: 8 min per run, 1 harness build; Hermes: 8 min per run, 1 harness build; OpenClaw: 30 min per run, 1 harness build; opencode: 30 min per run, 1 harness build.
One run per task
Every task ran once per cell, in one phrasing. Run-to-run variation is therefore not measured: a single task's difference can be one lucky or unlucky attempt. Every row of the reference table treats the TASKS as the unit: its intervals resample the tasks and its p-values come from a paired test over them, so they describe how much the result depends on which tasks were drawn — not how it would change if the same tasks were run again. With few tasks no difference can reach significance (five tasks cannot go below p = 1/16).
Sample size
13 cells × 69 tasks × 1 group = 897 runs.