Delivered · Study5views
WindTunnel A/B — Playwright vs WebMCP (15 tasks)
WebMCP vs Playwright · 15 tasks · 2 harnesses · 2 models
The benchmark pass rate favors WebMCP, but efficiency gains are offset by higher peak costs and no independent quality signal.
Correctness did not separate the arms beyond the decision margin, and the treatment got there with 36% fewer turns and 35% faster.
Abstract
WindTunnel tested 15 tasks across 2 harness/model cells, comparing WebMCP and Playwright setups. The benchmark itself found WebMCP at 100% pass rate against Playwright at 90%, a +10pp (+11.1% relative) advantage. On efficiency, WebMCP showed mixed results: cost came in at $0.04 (mean $0.24) against Playwright at $0.04 (mean $0.19), a +13.7% cost increase, while duration improved to 4m 19s (mean 6m 28s) from 6m 42s (mean 7m 47s), a -35.5% reduction. Tokens were nearly flat at 169.7k (mean 1.1M) versus 176.7k (mean 904.6k), a -4% change. However, WebMCP's worst case is worse: cost up to $3.50 against $2.33, and tokens up to 16M against 10.9M. No judge evaluated the results, and harness and model effects cannot be separated.
The result
PlaywrightWebMCP
Best arm overall: WebMCP (92.3 vs 82.7 of 100)
Best setup: claude-code / claude-haiku-4-5-20251001 · WebMCP (95.2 of 100)
- 1claude-code / claude-haiku-4-5-20251001WebMCP95.2best
- 2claude-code / claude-haiku-4-5-20251001Playwright91.4
- 3codex / gpt-5.4-miniWebMCP89.3
- 4codex / gpt-5.4-miniPlaywright76.3
Overall score per arm, 0–100 points (not a pass rate): 75% benchmark pass rate (the benchmark's own graders carry the outcome share — no judge grades or evals on this board) + 25% efficiency (cost · tokens · duration, vs the board's best arm). Absent components renormalize. Best at the top — the board's best setup is tagged.
Quality × efficiency clusters — normalized per task
Every graded run, standardized WITHIN its task so difficulty cancels out: → right = fewer tokens than the field on the same task, ↑ up = higher quality index (85% the benchmark's own pass/fail + 15% outcome) than the field. The field pools every harness and model, so a setup's left–right position largely reflects its own token habits. Dots are runs; each harness logo is a harness × model × arm centroid — hover it for the model and averages. Up-right wins.
2 runs not plotted (missing a grade or token count — infra failures included).
Every metric, per harness × model
Benchmark pass rate
- claude-code / claude-haiku-4-5-20251001100% → 100%even
- codex / gpt-5.4-mini79% → 100%21.4 pp better
Task completion
- claude-code / claude-haiku-4-5-20251001100% → 100%even
- codex / gpt-5.4-mini93% → 100%6.7 pp better
Cost per run
- claude-code / claude-haiku-4-5-20251001$0.05 → $0.0344% better
- codex / gpt-5.4-mini$0.04 → $0.0679% worse
Tokens per run
- claude-code / claude-haiku-4-5-20251001255.6k → 176.6k31% better
- codex / gpt-5.4-mini105.1k → 234.7k123% worse
Duration per run
- codex / gpt-5.4-mini10m 21s → 4m 22s58% better
- claude-code / claude-haiku-4-5-202510013m 45s → 4m 31s20% worse
Playwright
WebMCP
The publisher has not described this setup — what each group's tools are, who built them, and how the tasks were chosen.
Distributions
Every completed run is one dot — the spread the averages hide. Click a dot to replay that run's journey.
Cost per run
Token composition — average per run
Run outcomes
How each run ended — whether the agent finished, not whether its answer passed. Whether answers passed is in the results above.
Statistics
| Metric | Arm | n | Mean | Median (pooled) | Min | Max | Std dev |
|---|---|---|---|---|---|---|---|
| Cost | base | 29/30 | $0.19 | $0.04 | $0.01 | $2.33 | $0.46 |
| treat | 30/30 | $0.24 | $0.03 | $0.02 | $3.50 | $0.72 | |
| Duration | base | 30/30 | 7m 47s | 7m 29s | 2m 05s | 31m 37s | 6m 02s |
| treat | 30/30 | 6m 28s | 4m 26s | 2m 04s | 18m 05s | 4m 15s | |
| Total tokens | base | 29/30 | 904.6k | 236.6k | 35k | 10.9M | 2.1M |
| treat | 30/30 | 1.1M | 195.7k | 34.5k | 16M | 3.3M | |
| Output tokens | base | 29/30 | 4.4k | 1.3k | 241 | 40.6k | 8.4k |
| treat | 30/30 | 4.9k | 1k | 398 | 54.2k | 11.9k | |
| Cache-read tokens | base | 29/30 | 870.3k | 228.5k | 21.8k | 10.7M | 2.1M |
| treat | 30/30 | 1.1M | 185.7k | 21.2k | 15.5M | 3.2M | |
| Cache-write tokens | base | 29/30 | 8.3k | 4.3k | 0 | 42.2k | 12.7k |
| treat | 30/30 | 3.5k | 1.7k | 0 | 25.7k | 5.3k | |
| Turns | base | 29/30 | 14.3 | 8 | 2 | 56 | 15.4 |
| treat | 30/30 | 12.8 | 8 | 3 | 50 | 11.5 |
n = runs carrying the fact / completed runs in the arm; every statistic runs over present facts only — a missing fact is never counted as 0. Std dev is the sample form (n−1), withheld below n = 2. Every figure here POOLS all runs. The typical-run figures in the abstract, the head-to-head decision and the per-setup panels take each task's median first and then the median across tasks, so a task that ran more often never outweighs one that ran once; the two medians can differ. A promise study's headline (“cut average tokens”) compares per-arm means.
What each setup did
Derived from each run's recorded tool calls — not from a model's description of the run — and aggregated per setup, so a behavior seen across several runs is stated once with its rate. Open a finding to see the runs behind it, each linked to its journey at the step where it happened.
codex · openai · gpt-5.4-mini · Playwright4 findings
- 2 of 15 runs · 1 completed
- 2 of 15 runs · 1 completed
- 2 of 15 runs · 1 completed
- 1 of 15 runs · 0 completed
codex · openai · gpt-5.4-mini · WebMCP3 findings
- 3 of 15 runs · 3 completed
- 3 of 15 runs · 3 completed
- 2 of 15 runs · 2 completed
Task text withheld
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Tasks
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Task text withheld — this benchmark is guarded and its tasks are not republished here.
Runs
Every run is inspectable — open one to replay the agent's journey step by step, with the analyst's read underneath.
Every run, filterable… of 60Show runsHide runs
| Harness | Model | Task | Arm | Completed | Pass | Tokens | Cost | Duration |
|---|
Loading runs…
Methodology
What each metric means
- Completed
- The run finished and the harness returned a response. It does NOT mean the answer was correct — a completed run can score zero on quality.
- Denominator: Terminal runs, excluding those killed by our own infrastructure.
- Quality index
- A composite used only in the quality x efficiency plot, built from whichever grader scored this study's runs — the judge's rubric, the pre-registered checks or the benchmark's own verdict — plus whether the run's own outcome was success. The map's caption names the grader and its weights.
- Denominator: Graded runs — runs the study's grader scored.
- Infra-excluded
- A run killed by our own infrastructure. It is a missing measurement, never a loss for the arm, and is excluded from every rate denominator.
- Denominator: Reported as a count beside every affected panel.
- A/B arms
- Every task runs twice per harness × model cell — once as Playwright, once as WebMCP — on the same prompt, same model, cold start for both arms. Both arms are real configurations.
- One run per task
- Every task ran once per cell and arm, in one phrasing. Run-to-run variation is therefore not measured: a single task's difference can be one lucky or unlucky attempt. Every row of the reference table treats the TASKS as the unit: its intervals resample the tasks and its p-values come from a paired test over them, so they describe how much the result depends on which tasks were drawn — not how it would change if the same tasks were run again. With few tasks no difference can reach significance (five tasks cannot go below p = 1/16).
- Sample size
- 2 cells × 15 tasks × 2 arms = 60 runs.