Measured · Study2views
kalanu index crossed block — HumanEval — claude-agent-sdk and opencode on claude-sonnet-5-5
HumanEval · 164 tasks · 2 harnesses · 1 model
Opencode on claude-sonnet-5-5 achieved the highest pass rate on HumanEval.
Ranked leaderboard of agents under As published: cells ranked by the composite score — the benchmark's own graders, then efficiency — opencode / claude-sonnet-5-5 leads.
From the publisher
kalanu.ai crossed block (owner-approved D5, 2026-10-03). The full HumanEval corpus (164 tasks), the same pinned version as the 13-setup seed study, run once through the two setups that complete the block {claude-code, claude-agent-sdk, opencode} x {claude-haiku-4-5, claude-sonnet-5-5}: claude-agent-sdk and opencode on claude-sonnet-5-5 at their default configuration. The other four setups of the block were run in the seed study. Graded by the benchmark's own grader. One repeat per task.
Abstract
The opencode harness paired with claude-sonnet-5-5 led the leaderboard, with runs passing HumanEval's 164 tasks at a rate of 98%. Efficiency metrics show a median cost of $0.01 (mean $0.01) and median token consumption of 20k (mean 20k), with a median duration of 10.6s (mean 10.7s). This single-model benchmark measured 2 harness/model cells under one group, As published.
The result
claude-agent-sdkopencode
Best setup: opencode / Claude Sonnet 5.5 (91.6 of 100)
- 1opencode / Claude Sonnet 5.591.6best
- 2Claude Agent SDK / Claude Sonnet 5.587.8
Overall score per cell, 0–100 points (not a pass rate): 75% benchmark pass rate (the benchmark's own graders carry the outcome share — no judge grades or evals on this board) + 25% efficiency (cost · tokens · duration, vs the board's best agent). Absent components renormalize. Best at the top — the board's best setup is tagged.
Leaderboard
Cells ranked by the composite score — outcome (the lenses that graded) × efficiency — absolute values, no other group. Tokens, cost and duration are per-cell medians, the efficiency the score folds.
| # | Harness | Model | Score | Pass rate | Quality | Evals | Goal | Tokens | Cost | Duration | Runs |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | opencode | Claude Sonnet 5.5 | 91.6 | 98% | − | − | − | 11k | $0.0048 | 18.3s | 164/164 |
| 2 | Claude Agent SDK | Claude Sonnet 5.5 | 87.8 | 98% | − | − | − | 28.9k | $0.02 | 3s | 164/164 |
Quality × efficiency clusters — normalized per task
Every graded run, standardized WITHIN its task so difficulty cancels out: → right = fewer tokens than the field on the same task, ↑ up = higher quality index (85% the benchmark's own pass/fail + 15% outcome) than the field. Within every task, all runs scored alike on quality, so the vertical axis separates nothing — read the map left to right. The field pools every harness and model, so a setup's left–right position largely reflects its own token habits. Dots are runs; each harness logo is a harness × model × group centroid — hover it for the model and averages. Up-right wins.
Every metric, per harness × model
Benchmark pass rate
- Claude Agent SDK / Claude Sonnet 5.598%
- opencode / Claude Sonnet 5.598%
Task completion
- Claude Agent SDK / Claude Sonnet 5.5100%
- opencode / Claude Sonnet 5.5100%
Cost per run
- opencode / Claude Sonnet 5.5$0.0048
- Claude Agent SDK / Claude Sonnet 5.5$0.02
Tokens per run
- opencode / Claude Sonnet 5.511k
- Claude Agent SDK / Claude Sonnet 5.528.9k
Duration per run
- Claude Agent SDK / Claude Sonnet 5.53s
- opencode / Claude Sonnet 5.518.3s
By model
By model
Benchmark pass rate
By model
- Claude Sonnet 5.598%
Task completion
By model
- Claude Sonnet 5.5100%
Cost per run
By model
- Claude Sonnet 5.5$0.01
Tokens per run
By model
- Claude Sonnet 5.520k
Duration per run
By model
- Claude Sonnet 5.510.6s
Distributions
Every completed run is one dot — the spread the averages hide. Click a dot to replay that run's journey.
Cost per run
Token composition — average per run
Statistics
| Metric | Group | n | Mean | Median (pooled) | Min | Max | Std dev |
|---|---|---|---|---|---|---|---|
| Cost | As published | 328/328 | $0.01 | $0.01 | $0.0031 | $0.07 | $0.01 |
| Duration | As published | 328/328 | 10.7s | 11.7s | 2.2s | 24.4s | 7.6s |
| Total tokens | As published | 328/328 | 20k | 20.2k | 10.7k | 29.6k | 9k |
| Output tokens | As published | 328/328 | 203 | 184 | 31 | 783 | 109 |
| Cache-read tokens | As published | 328/328 | 17.6k | 10.5k | 0 | 25.4k | 7.8k |
| Cache-write tokens | As published | 328/328 | 2.2k | 3.3k | 195 | 28.7k | 3.1k |
| Turns | As published | 164/328 | 1 | 1 | 1 | 1 | 0 |
n = runs carrying the fact / completed runs in the group; every statistic runs over present facts only — a missing fact is never counted as 0. Std dev is the sample form (n−1), withheld below n = 2. Every figure here POOLS all runs. The typical-run figures in the abstract, the head-to-head decision and the per-setup panels take each task's median first and then the median across tasks, so a task that ran more often never outweighs one that ran once; the two medians can differ. A promise study's headline (“cut average tokens”) compares per-group means.
What each setup did
Derived from each run's recorded tool calls — not from a model's description of the run — and aggregated per setup, so a behavior seen across several runs is stated once with its rate. Open a finding to see the runs behind it, each linked to its journey at the step where it happened.
Claude Agent SDK · anthropic · Claude Sonnet 5.5 · As published1 finding
- 164 of 164 runs · 164 completed
opencode · anthropic · Claude Sonnet 5.5 · As published1 finding
- 164 of 164 runs · 164 completed
Task text withheld
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Tasks
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Runs
Every run is inspectable — open one to replay the agent's journey step by step, with the analyst's read underneath.
Every run, filterable… of 328Show runsHide runs
| Harness | Model | Task | Completed | Pass | Tokens | Cost | Duration |
|---|
Loading runs…
Methodology
What each metric means
- Completed
- The run finished and the harness returned a response. It does NOT mean the answer was correct — a completed run can score zero on quality.
- Denominator: Terminal runs, excluding those killed by our own infrastructure.
- Quality index
- A composite used only in the quality x efficiency plot, built from whichever grader scored this study's runs — the judge's rubric, the pre-registered checks or the benchmark's own verdict — plus whether the run's own outcome was success. The map's caption names the grader and its weights.
- Denominator: Graded runs — runs the study's grader scored.
- Infra-excluded
- A run killed by our own infrastructure. It is a missing measurement, never a loss for the group, and is excluded from every rate denominator.
- Denominator: Reported as a count beside every affected panel.
- One group
- Every task runs once per harness × model cell — a leaderboard with no other group. The readout is absolute (quality, success, tokens, cost) and the cells are ranked into a leaderboard.
- Run budget and harness builds
- Each run had a wall-clock budget and ran on a recorded harness build — disclosed, never used to split a comparison: Claude Agent SDK: 8 min per run, 1 harness build; opencode: 30 min per run, 1 harness build.
- One run per task
- Every task ran once per cell, in one phrasing. Run-to-run variation is therefore not measured: a single task's difference can be one lucky or unlucky attempt. Every row of the reference table treats the TASKS as the unit: its intervals resample the tasks and its p-values come from a paired test over them, so they describe how much the result depends on which tasks were drawn — not how it would change if the same tasks were run again. With few tasks no difference can reach significance (five tasks cannot go below p = 1/16).
- Sample size
- 2 cells × 164 tasks × 1 group = 328 runs.