Measured · Study5views
Seed pilot — AIME 2025 · claude-code / Haiku 4.5
AIME 2025 · 30 tasks · 1 harness · 1 model
The As published setup achieved a pass rate of 84% on AIME 2025 with a median cost of $0.10.
One agent under As published: claude-code / claude-haiku-4-5-20251001 is the only setup, so there is no ranking — its figures are an absolute readout.
From the publisher
kalanu Index seed pilot: measures cost and time per run on the full AIME 2025 corpus under harness default (no add-ons), at the recommended 4 repeats.
Abstract
The As published setup passed 84% of the 30 AIME 2025 tasks. This single harness and model cell represents an absolute readout of performance on the benchmark's published task set, with no second group to establish ranking.
Efficiency metrics show a median cost of $0.10 (mean $0.14), median token count of 52.4k (mean 152k), and median duration of 1m 56s (mean 3m 04s) across the tasks. These figures reflect the resource requirements for the As published setup's run.
The result
claude-code
- 1Claude Code / Claude Haiku 4.588.1
Overall score per cell, 0–100 points (not a pass rate): 75% benchmark pass rate (the benchmark's own graders carry the outcome share — no judge grades or evals on this board) + 25% efficiency (cost · tokens · duration, vs the board's best agent). Absent components renormalize. Best at the top — the board's best setup is tagged.
Leaderboard
Cells ranked by the composite score — outcome (the lenses that graded) × efficiency — absolute values, no other group. Tokens, cost and duration are per-cell medians, the efficiency the score folds.
| # | Harness | Model | Score | Pass rate | Quality | Evals | Goal | Tokens | Cost | Duration | Runs |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Code | Claude Haiku 4.5 | 88.1 | 84% | − | − | − | 52.4k | $0.10 | 1m 56s | 120/120 |
Pass rate
84.2% (95% CI 74.2–92.5)over 120 graded runs| Cell | Pass rate | Tasks | Runs | Ran as | Matcher |
|---|---|---|---|---|---|
| claude-code/claude-haiku-4-5-20251001 | 84.2% (95% CI 75–92.5) | 30 | 120 | harness default | numeric |
Each run's answer was extracted from its response and matched against the benchmark's own key — no model in the loop. Two-stage: the mean over repeats per task, then the mean over tasks, with the interval taken over tasks.
Quality × efficiency clusters — normalized per task
Every graded run, standardized WITHIN its task so difficulty cancels out: → right = fewer tokens than the field on the same task, ↑ up = higher quality index (85% the benchmark's own pass/fail + 15% outcome) than the field. The field pools every harness and model, so a setup's left–right position largely reflects its own token habits. Dots are runs; each harness logo is a harness × model × group centroid — hover it for the model and averages. Up-right wins.
Claude Code
Every metric, per harness × model
Benchmark pass rate
- Claude Code / Claude Haiku 4.584%
Task completion
- Claude Code / Claude Haiku 4.5100%
Cost per run
- Claude Code / Claude Haiku 4.5$0.10
Tokens per run
- Claude Code / Claude Haiku 4.552.4k
Duration per run
- Claude Code / Claude Haiku 4.51m 56s
Distributions
Every completed run is one dot — the spread the averages hide. Click a dot to replay that run's journey.
Cost per run
Token composition — average per run
Statistics
| Metric | Group | n | Mean | Median (pooled) | Min | Max | Std dev |
|---|---|---|---|---|---|---|---|
| Cost | As published | 120/120 | $0.14 | $0.10 | $0.01 | $0.90 | $0.14 |
| Duration | As published | 120/120 | 3m 04s | 1m 55s | 20.3s | 20m 13s | 3m 19s |
| Total tokens | As published | 120/120 | 152k | 52.1k | 35.3k | 2.4M | 326k |
| Output tokens | As published | 120/120 | 23.6k | 17.6k | 1.2k | 119.9k | 21.4k |
| Cache-read tokens | As published | 120/120 | 122.3k | 34.2k | 0 | 2.2M | 299.8k |
| Cache-write tokens | As published | 120/120 | 6.1k | 0 | 0 | 66.4k | 13.1k |
| Turns | As published | 120/120 | 5.6 | 2 | 2 | 74 | 11.6 |
n = runs carrying the fact / completed runs in the group; every statistic runs over present facts only — a missing fact is never counted as 0. Std dev is the sample form (n−1), withheld below n = 2. Every figure here POOLS all runs. The typical-run figures in the abstract, the head-to-head decision and the per-setup panels take each task's median first and then the median across tasks, so a task that ran more often never outweighs one that ran once; the two medians can differ. A promise study's headline (“cut average tokens”) compares per-group means.
What each setup did
Derived from each run's recorded tool calls — not from a model's description of the run — and aggregated per setup, so a behavior seen across several runs is stated once with its rate. Open a finding to see the runs behind it, each linked to its journey at the step where it happened.
Claude Code · anthropic · Claude Haiku 4.5 · As published3 findings
- 105 of 120 runs · 105 completed
- 8 of 120 runs · 8 completed
- 4 of 120 runs · 4 completed
Task text withheld
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Tasks
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Runs
Every run is inspectable — open one to replay the agent's journey step by step, with the analyst's read underneath.
Every run, filterable… of 120Show runsHide runs
| Harness | Model | Task | Repeat | Completed | Pass | Tokens | Cost | Duration |
|---|
Loading runs…
Methodology
What each metric means
- Completed
- The run finished and the harness returned a response. It does NOT mean the answer was correct — a completed run can score zero on quality.
- Denominator: Terminal runs, excluding those killed by our own infrastructure.
- Quality index
- A composite used only in the quality x efficiency plot, built from whichever grader scored this study's runs — the judge's rubric, the pre-registered checks or the benchmark's own verdict — plus whether the run's own outcome was success. The map's caption names the grader and its weights.
- Denominator: Graded runs — runs the study's grader scored.
- Infra-excluded
- A run killed by our own infrastructure. It is a missing measurement, never a loss for the group, and is excluded from every rate denominator.
- Denominator: Reported as a count beside every affected panel.
- One group
- Every task runs once per harness × model cell — a leaderboard with no other group. The readout is absolute (quality, success, tokens, cost) and the cells are ranked into a leaderboard.
- Sampling
- Each task is asked in 1 phrasing, and every phrasing is run 4 times per group. Every one of those runs is a separate sample, graded against the same frozen rubric — so a result that only holds for one wording, or on one lucky attempt, shows up as disagreement between samples instead of passing for a finding.
- Sample size
- 30 tasks × 1 variant × 4 repeats × 1 cell × 1 group = 120 runs.