Measured · Study2views
kalanu index seed — WindTunnel — 13 harness setups
WindTunnel · 50 tasks · 11 harnesses · 4 models
Claude-code / claude-sonnet-5-5 leads the WindTunnel leaderboard.
Ranked leaderboard of agents under As published: cells ranked by the composite score — the benchmark's own graders, then efficiency — claude-code / claude-sonnet-5-5 leads.
From the publisher
kalanu.ai index seed (owner-approved T1 tier, 2026-10-01). The full WindTunnel corpus run once through each of 13 agent harnesses at their default configuration (twelve harnesses on their default model plus two frontier rows), graded by the benchmark's own reference answers or verifier. One repeat per task. nanoclaw joins in a follow-up study once its runtime fix is live.
Abstract
WindTunnel runs passed their tasks at a rate of 63% across the As published leaderboard of 13 harness/model cells. The leader is claude-code / claude-sonnet-5-5. Efficiency metrics show a median cost of $0.13 (mean $0.24), median tokens of 601.1k (mean 1.1M), and median duration of 1m 10s (mean 2m 10s) across the 50 tasks. This benchmark evaluated a single group, As published, with no second group for comparison.
The result
aiderclaude-agent-sdkclaude-codecodexcontinuedeepagentsdeepseekevehermesopenclawopencode
Best setup: Claude Code / Claude Sonnet 5.5 (76.9 of 100)
- 1Claude Code / Claude Sonnet 5.576.9best
- 2Codex / GPT-6.1 Sol74.2
- 3Codex / GPT-5.4 Mini68.9
- 4Hermes / Claude Haiku 4.558.9
- 5DeepSeek Harness / Claude Haiku 4.555.5
- 6eve / Claude Haiku 4.553.7
- 7Aider / Claude Haiku 4.551.5
- 8Claude Agent SDK / Claude Haiku 4.550.5
- 9Continue / Claude Haiku 4.546.7
- 10opencode / Claude Haiku 4.545.1
- 11Claude Code / Claude Haiku 4.542.9
- 12OpenClaw / Claude Haiku 4.529.3
- 13DeepAgents / Claude Haiku 4.516.6
Overall score per cell, 0–100 points (not a pass rate): 75% benchmark pass rate (the benchmark's own graders carry the outcome share — no judge grades or evals on this board) + 25% efficiency (cost · tokens · duration, vs the board's best agent). Absent components renormalize. Best at the top — the board's best setup is tagged.
Leaderboard
Cells ranked by the composite score — outcome (the lenses that graded) × efficiency — absolute values, no other group. Tokens, cost and duration are per-cell medians, the efficiency the score folds.
| # | Harness | Model | Score | Pass rate | Quality | Evals | Goal | Tokens | Cost | Duration | Runs |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Code | Claude Sonnet 5.5 | 76.9 | 98% | − | − | − | 157.8k | $0.06 | 28.4s | 50/50 |
| 2 | Codex | GPT-6.1 Sol | 74.2 | 98% | − | − | − | 670.1k | $0.16 | 2m 50s | 50/50 |
| 3 | Codex | GPT-5.4 Mini | 68.9 | 90% | − | − | − | 481.1k | $0.10 | 57.7s | 50/50 |
| 4 | Hermes | Claude Haiku 4.5 | 58.9 | 78% | − | − | − | 1.2M | $0.24 | 2m 15s | 49/50 |
| 5 | DeepSeek Harness | Claude Haiku 4.5 | 55.5 | 73% | − | − | − | 1.7M | $0.32 | 1m 52s | 49/50 |
| 6 | eve | Claude Haiku 4.5 | 53.7 | 70% | − | − | − | 479.9k | $0.13 | 1m 51s | 50/50 |
| 7 | Aider | Claude Haiku 4.5 | 51.5 | 43% | − | − | − | 3k | $0.0037 | 28.4s | 50/50 |
| 8 | Claude Agent SDK | Claude Haiku 4.5 | 50.5 | 65% | − | − | − | 317.1k | $0.06 | 1m 21s | 50/50 |
| 9 | Continue | Claude Haiku 4.5 | 46.7 | 60% | − | − | − | 113.6k | $0.12 | 48.8s | 49/50 |
| 10 | opencode | Claude Haiku 4.5 | 45.1 | 58% | − | − | − | 241.2k | $0.07 | 1m 11s | 50/50 |
| 11 | Claude Code | Claude Haiku 4.5 | 42.9 | 54% | − | − | − | 357.9k | $0.06 | 41.5s | 50/50 |
| 12 | OpenClaw | Claude Haiku 4.5 | 29.3 | 37% | − | − | − | 395.1k | $0.07 | 1m 12s | 50/50 |
| 13 | DeepAgents | Claude Haiku 4.5 | 16.6 | 8% | − | − | − | 31.4k | $0.02 | 9.3s | 50/50 |
Pass rate
67.7% (95% CI 58.9–75.1)over 537 graded runs- Task text is withheld from public pages until an admin verifies this licence. Running this benchmark is unaffected.
| Cell | Pass rate | Tasks | Runs | Ran as | Matcher |
|---|---|---|---|---|---|
| aider/claude-haiku-4-5-20251001 | 51.2% (95% CI 36.6–65.9) | 41 | 41 (+9 unmeasured) | harness default | text_predicate |
| claude-agent-sdk/claude-haiku-4-5-20251001 | 71.4% (95% CI 57.1–83.3) | 42 | 42 (+8 unmeasured) | harness default | text_predicate |
| claude-code/claude-haiku-4-5-20251001 | 59.5% (95% CI 45.2–73.8) | 42 | 42 (+8 unmeasured) | harness default | text_predicate |
| claude-code/claude-sonnet-5-5 | 97.6% (95% CI 92.9–100) | 42 | 42 (+8 unmeasured) | harness default | text_predicate |
| codex/gpt-5.4-mini | 90.2% (95% CI 80.5–97.6) | 41 | 41 (+9 unmeasured) | harness default | text_predicate |
| codex/gpt-6.1-sol | 97.6% (95% CI 92.9–100) | 42 | 42 (+8 unmeasured) | harness default | text_predicate |
| continue/claude-haiku-4-5-20251001 | 63.4% (95% CI 48.8–78.1) | 41 | 41 (+9 unmeasured) | harness default | text_predicate |
| deepagents/claude-haiku-4-5-20251001 | 9.5% (95% CI 2.4–21.4) | 42 | 42 (+8 unmeasured) | harness default | text_predicate |
| deepseek/claude-haiku-4-5-20251001 | 80.5% (95% CI 68.3–92.7) | 41 | 41 (+9 unmeasured) | harness default | text_predicate |
| eve/claude-haiku-4-5-20251001 | 75.6% (95% CI 63.4–87.8) | 41 | 41 (+9 unmeasured) | harness default | text_predicate |
| hermes/claude-haiku-4-5-20251001 | 82.9% (95% CI 70.7–92.7) | 41 | 41 (+9 unmeasured) | harness default | text_predicate |
| openclaw/claude-haiku-4-5-20251001 | 38.5% (95% CI 23.1–53.9) | 39 | 39 (+11 unmeasured) | harness default | text_predicate |
| opencode/claude-haiku-4-5-20251001 | 64.3% (95% CI 50–78.6) | 42 | 42 (+8 unmeasured) | harness default | text_predicate |
Each run's answer was extracted from its response and matched against the benchmark's own key — no model in the loop. Two-stage: the mean over repeats per task, then the mean over tasks, with the interval taken over tasks.
Quality × efficiency clusters — normalized per task
Every graded run, standardized WITHIN its task so difficulty cancels out: → right = fewer tokens than the field on the same task, ↑ up = higher quality index (85% the benchmark's own pass/fail + 15% outcome) than the field. The field pools every harness and model, so a setup's left–right position largely reflects its own token habits. Dots are runs; each harness logo is a harness × model × group centroid — hover it for the model and averages. Up-right wins.
18 runs not plotted (missing a grade or token count — infra failures included).
Every metric, per harness × model
Benchmark pass rate
- Codex / GPT-6.1 Sol98%
- Claude Code / Claude Sonnet 5.598%
- Codex / GPT-5.4 Mini90%
- Hermes / Claude Haiku 4.578%
- DeepSeek Harness / Claude Haiku 4.573%
- eve / Claude Haiku 4.570%
- Claude Agent SDK / Claude Haiku 4.565%
- Continue / Claude Haiku 4.560%
- opencode / Claude Haiku 4.558%
- Claude Code / Claude Haiku 4.554%
- Aider / Claude Haiku 4.543%
- OpenClaw / Claude Haiku 4.537%
- DeepAgents / Claude Haiku 4.58%
Task completion
- Aider / Claude Haiku 4.5100%
- Claude Agent SDK / Claude Haiku 4.5100%
- Claude Code / Claude Haiku 4.5100%
- Claude Code / Claude Sonnet 5.5100%
- Continue / Claude Haiku 4.5100%
- DeepAgents / Claude Haiku 4.5100%
- DeepSeek Harness / Claude Haiku 4.5100%
- Hermes / Claude Haiku 4.5100%
- opencode / Claude Haiku 4.5100%
- Codex / GPT-6.1 Sol98%
- Codex / GPT-5.4 Mini96%
- eve / Claude Haiku 4.594%
- OpenClaw / Claude Haiku 4.592%
Cost per run
- Aider / Claude Haiku 4.5$0.0037
- DeepAgents / Claude Haiku 4.5$0.02
- Claude Agent SDK / Claude Haiku 4.5$0.06
- Claude Code / Claude Sonnet 5.5$0.06
- Claude Code / Claude Haiku 4.5$0.06
- OpenClaw / Claude Haiku 4.5$0.07
- opencode / Claude Haiku 4.5$0.07
- Codex / GPT-5.4 Mini$0.10
- Continue / Claude Haiku 4.5$0.12
- eve / Claude Haiku 4.5$0.13
- Codex / GPT-6.1 Sol$0.16
- Hermes / Claude Haiku 4.5$0.24
- DeepSeek Harness / Claude Haiku 4.5$0.32
Tokens per run
- Aider / Claude Haiku 4.53k
- DeepAgents / Claude Haiku 4.531.4k
- Continue / Claude Haiku 4.5113.6k
- Claude Code / Claude Sonnet 5.5157.8k
- opencode / Claude Haiku 4.5241.2k
- Claude Agent SDK / Claude Haiku 4.5317.1k
- Claude Code / Claude Haiku 4.5357.9k
- OpenClaw / Claude Haiku 4.5395.1k
- eve / Claude Haiku 4.5479.9k
- Codex / GPT-5.4 Mini481.1k
- Codex / GPT-6.1 Sol670.1k
- Hermes / Claude Haiku 4.51.2M
- DeepSeek Harness / Claude Haiku 4.51.7M
Duration per run
- DeepAgents / Claude Haiku 4.59.3s
- Aider / Claude Haiku 4.528.4s
- Claude Code / Claude Sonnet 5.528.4s
- Claude Code / Claude Haiku 4.541.5s
- Continue / Claude Haiku 4.548.8s
- Codex / GPT-5.4 Mini57.7s
- opencode / Claude Haiku 4.51m 11s
- OpenClaw / Claude Haiku 4.51m 12s
- Claude Agent SDK / Claude Haiku 4.51m 21s
- eve / Claude Haiku 4.51m 51s
- DeepSeek Harness / Claude Haiku 4.51m 52s
- Hermes / Claude Haiku 4.52m 15s
- Codex / GPT-6.1 Sol2m 50s
By harness, by model
By harness
By model
Benchmark pass rate
By harness
- Codex94%
- Hermes78%
- Claude Code75%
- DeepSeek Harness73%
- eve70%
- Claude Agent SDK65%
- Continue60%
- opencode58%
- Aider43%
- OpenClaw37%
- DeepAgents8%
By model
- GPT-6.1 Sol98%
- Claude Sonnet 5.598%
- GPT-5.4 Mini90%
- Claude Haiku 4.554%
Task completion
By harness
- Aider100%
- Claude Agent SDK100%
- Claude Code100%
- Continue100%
- DeepAgents100%
- DeepSeek Harness100%
- Hermes100%
- opencode100%
- Codex97%
- eve94%
- OpenClaw92%
By model
- Claude Sonnet 5.5100%
- Claude Haiku 4.599%
- GPT-6.1 Sol98%
- GPT-5.4 Mini96%
Cost per run
By harness
- Aider$0.0037
- DeepAgents$0.02
- Claude Agent SDK$0.06
- OpenClaw$0.07
- Claude Code$0.07
- opencode$0.07
- Continue$0.12
- eve$0.13
- Codex$0.15
- Hermes$0.24
- DeepSeek Harness$0.32
By model
- Claude Sonnet 5.5$0.06
- GPT-5.4 Mini$0.10
- Claude Haiku 4.5$0.12
- GPT-6.1 Sol$0.16
Tokens per run
By harness
- Aider3k
- DeepAgents31.4k
- Continue113.6k
- opencode241.2k
- Claude Code280.3k
- Claude Agent SDK317.1k
- OpenClaw395.1k
- eve479.9k
- Codex697.4k
- Hermes1.2M
- DeepSeek Harness1.7M
By model
- Claude Sonnet 5.5157.8k
- GPT-5.4 Mini481.1k
- Claude Haiku 4.5583k
- GPT-6.1 Sol670.1k
Duration per run
By harness
- DeepAgents9.3s
- Aider28.4s
- Claude Code37.7s
- Continue48.8s
- opencode1m 11s
- OpenClaw1m 12s
- Claude Agent SDK1m 21s
- eve1m 51s
- DeepSeek Harness1m 52s
- Codex2m 00s
- Hermes2m 15s
By model
- Claude Sonnet 5.528.4s
- GPT-5.4 Mini57.7s
- Claude Haiku 4.51m 24s
- GPT-6.1 Sol2m 50s
Distributions
Every completed run is one dot — the spread the averages hide. Click a dot to replay that run's journey.
Cost per run
Token composition — average per run
Statistics
| Metric | Group | n | Mean | Median (pooled) | Min | Max | Std dev |
|---|---|---|---|---|---|---|---|
| Cost | As published | 647/650 | $0.24 | $0.06 | $0.0027 | $5.42 | $0.48 |
| Duration | As published | 650/650 | 2m 10s | 50.7s | 1.8s | 30m 00s | 3m 44s |
| Total tokens | As published | 647/650 | 1.1M | 214.9k | 2.5k | 31.9M | 2.2M |
| Output tokens | As published | 647/650 | 6k | 1.5k | 20 | 134.8k | 10.3k |
| Cache-read tokens | As published | 647/650 | 996.4k | 157.1k | 0 | 29.5M | 2.1M |
| Cache-write tokens | As published | 647/650 | 22.7k | 6.5k | 0 | 1.3M | 59.9k |
| Turns | As published | 397/650 | 30.1 | 8 | 1 | 422 | 48.5 |
n = runs carrying the fact / completed runs in the group; every statistic runs over present facts only — a missing fact is never counted as 0. Std dev is the sample form (n−1), withheld below n = 2. Every figure here POOLS all runs. The typical-run figures in the abstract, the head-to-head decision and the per-setup panels take each task's median first and then the median across tasks, so a task that ran more often never outweighs one that ran once; the two medians can differ. A promise study's headline (“cut average tokens”) compares per-group means.
What each setup did
Derived from each run's recorded tool calls — not from a model's description of the run — and aggregated per setup, so a behavior seen across several runs is stated once with its rate. Open a finding to see the runs behind it, each linked to its journey at the step where it happened.
Claude Agent SDK · anthropic · Claude Haiku 4.5 · As published5 findings
- 17 of 50 runs · 17 completed
- 16 of 50 runs · 16 completed
- 14 of 50 runs · 14 completed
- 7 of 50 runs · 7 completed
- 3 of 50 runs · 3 completed
Claude Code · anthropic · Claude Haiku 4.5 · As published6 findings
- 11 of 50 runs · 11 completed
- 11 of 50 runs · 11 completed
- 11 of 50 runs · 11 completed
- 8 of 50 runs · 8 completed
- 4 of 50 runs · 4 completed
- 2 of 50 runs · 2 completed
Codex · openai · GPT-5.4 Mini · As published5 findings
- 4 of 50 runs · 3 completed
- 3 of 50 runs · 3 completed
- 2 of 50 runs · 1 completed
- 1 of 50 runs · 0 completed
- 1 of 50 runs · 1 completed
Codex · openai · GPT-6.1 Sol · As published5 findings
- 28 of 50 runs · 27 completed
- 22 of 50 runs · 21 completed
- 20 of 50 runs · 20 completed
- 16 of 50 runs · 15 completed
- 4 of 50 runs · 3 completed
Continue · anthropic · Claude Haiku 4.5 · As published5 findings
- 12 of 49 runs · 12 completed
- 12 of 49 runs · 12 completed
- 8 of 49 runs · 8 completed
- 2 of 49 runs · 2 completed
- 2 of 49 runs · 2 completed
DeepSeek Harness · anthropic · Claude Haiku 4.5 · As published5 findings
- 21 of 49 runs · 21 completed
- 21 of 49 runs · 21 completed
- 21 of 49 runs · 21 completed
- 8 of 49 runs · 8 completed
- 7 of 49 runs · 7 completed
eve · anthropic · Claude Haiku 4.5 · As published5 findings
- 20 of 50 runs · 17 completed
- 18 of 50 runs · 15 completed
- 16 of 50 runs · 15 completed
- 5 of 50 runs · 4 completed
- 4 of 50 runs · 4 completed
Hermes · anthropic · Claude Haiku 4.5 · As published4 findings
- 15 of 49 runs · 15 completed
- 15 of 49 runs · 15 completed
- 12 of 49 runs · 12 completed
- 5 of 49 runs · 5 completed
OpenClaw · anthropic · Claude Haiku 4.5 · As published4 findings
- 8 of 50 runs · 7 completed
- 3 of 50 runs · 3 completed
- 3 of 50 runs · 3 completed
- 2 of 50 runs · 2 completed
opencode · anthropic · Claude Haiku 4.5 · As published5 findings
- 17 of 50 runs · 17 completed
- 17 of 50 runs · 17 completed
- 11 of 50 runs · 11 completed
- 3 of 50 runs · 3 completed
- 3 of 50 runs · 3 completed
Did each setup use its subject?
Read from each run's recorded steps. Runs on a harness whose evidence channel is not proven, and runs whose trace is incomplete, are not measured — never counted as unused.
- DeepSeek Harness · anthropic · Claude Haiku 4.5 · As published16 of 49 measured runs (33%) · 1 not measured
Task text withheld
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Tasks
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Runs
Every run is inspectable — open one to replay the agent's journey step by step, with the analyst's read underneath.
Every run, filterable… of 650Show runsHide runs
| Harness | Model | Task | Completed | Pass | Tokens | Cost | Duration |
|---|
Loading runs…
Methodology
What each metric means
- Completed
- The run finished and the harness returned a response. It does NOT mean the answer was correct — a completed run can score zero on quality.
- Denominator: Terminal runs, excluding those killed by our own infrastructure.
- Quality index
- A composite used only in the quality x efficiency plot, built from whichever grader scored this study's runs — the judge's rubric, the pre-registered checks or the benchmark's own verdict — plus whether the run's own outcome was success. The map's caption names the grader and its weights.
- Denominator: Graded runs — runs the study's grader scored.
- Infra-excluded
- A run killed by our own infrastructure. It is a missing measurement, never a loss for the group, and is excluded from every rate denominator.
- Denominator: Reported as a count beside every affected panel.
- One group
- Every task runs once per harness × model cell — a leaderboard with no other group. The readout is absolute (quality, success, tokens, cost) and the cells are ranked into a leaderboard.
- Run budget and harness builds
- Each run had a wall-clock budget and ran on a recorded harness build — disclosed, never used to split a comparison: Aider: 30 min per run, 1 harness build; Claude Agent SDK: 30 min per run, 1 harness build; Claude Code: 30 min per run, 1 harness build; Codex: 30 min per run, 1 harness build; Continue: 30 min per run, 1 harness build; DeepAgents: 30 min per run, 1 harness build; DeepSeek Harness: 30 min per run, 1 harness build; eve: 30 min per run, 1 harness build; Hermes: 30 min per run, 1 harness build; OpenClaw: 30 min per run, 1 harness build; opencode: 30 min per run, 1 harness build.
- One run per task
- Every task ran once per cell, in one phrasing. Run-to-run variation is therefore not measured: a single task's difference can be one lucky or unlucky attempt. Every row of the reference table treats the TASKS as the unit: its intervals resample the tasks and its p-values come from a paired test over them, so they describe how much the result depends on which tasks were drawn — not how it would change if the same tasks were run again. With few tasks no difference can reach significance (five tasks cannot go below p = 1/16).
- Sample size
- 13 cells × 50 tasks × 1 group = 650 runs.