Version 0.1 · updated 22 Nov 2026
Widal.Code Benchmark
Cost, time and solve rate of Claude Code setups on real agent sessions. Each setup runs the same tasks, is scored by hidden tests, and is compared with the same model running with no tool.
- Tasks
- 13
- Hidden checks
- 148
- Sessions
- 523
- Baseline
- Fable 5.2, xhigh
- Harness
- Claude Code 2.1.330
Leaderboard
Side-by-side run, 21 November 2026. Every setup on Fable 5.2 at effort xhigh. Sorted by cost.
| # | Setup | Solved | Cost | Δ cost | 95% CI | Cheaper tasks | p | Wall time | Δ time | Output tokens | Turns |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Widal.Code 0.3.1 | 13/13 | $10.97 | −59% | −65% to −52% | 13/13 | <0.001 | 40 min | −43% | 150k | 162 |
| 2 | caveman 4.1.0 | 13/13 | $17.27 | −36% | −47% to −22% | 11/13 | 0.022 | 37 min | −47% | 226k | 190 |
| 3 | Headroom 0.37.0 | 13/13 | $21.58 | −19% | −31% to −9% | 11/13 | 0.022 | 53 min | −25% | 325k | 286 |
| 4 | RTK 1.2.1 | 13/13 | $24.48 | −9% | −22% to +3% | 9/13 | 0.267 | 65 min | −7% | 361k | 283 |
| Baseline no tool | 13/13 | $26.79 | 70 min | 395k | 251 |
- Solved: every hidden check passes and the project's existing tests still pass. All setups solved 148 of 148 checks.
- Cost: Claude Code's reported cost at Anthropic list prices, subagents included. Wall time: start to end of each session, summed.
- Δ against the baseline of the same run. 95% CI: interval of Δ cost from resampling the 13 tasks (10,000 draws). It covers the choice of tasks, not run-to-run variation. Cheaper tasks and p: per-task cost against the baseline, two-sided sign test.
- The baseline varied by 36% between four runs. Single-run differences under about 35% (Headroom, RTK) are not conclusive.
Widal.Code against baseline
Per model, from effort xhigh with no tool. Widal.Code's changes are the ones it proposed from the baseline's own sessions.
| Model | Runs per task | Baseline | Widal.Code | Δ cost | 95% CI | Cheaper tasks | p | Δ time | Solved |
|---|---|---|---|---|---|---|---|---|---|
| Fable 5.2 | 3 | $81.26 | $34.64 | −57% | −61% to −53% | 38/39 | <0.001 | −47%* | 39/39 |
| Sonnet 5.5 | 1 | $3.78 | $1.87 | −51% | −58% to −41% | 13/13 | <0.001 | −58% | 13/13 |
| Opus 5.5 | 1 | $9.23 | $4.41 | −52% | −57% to −47% | 13/13 | <0.001 | −44% | 13/13 |
- * Two side-by-side runs. The first Fable 5.2 run was not side by side and its times are excluded.
- Changes applied: effort high on all three models, subagents on Sonnet 5.5 on Fable 5.2 and Opus 5.5, Bash output capped at 10,000 characters on Fable 5.2. Not run on Haiku 5.5.
Ablation
Fable 5.2, one side-by-side run on 21 Nov 2026. Each effort change alone, against Widal.Code's three changes.
| Setup | Cost | Δ cost | Solved |
|---|---|---|---|
| Baseline, effort xhigh | $29.38 | 13/13 | |
| Effort high only | $13.11 | −55% | 13/13 |
| Widal.Code, all three changes | $12.10 | −59% | 13/13 |
| Effort medium only | $8.62 | −71% | 13/13 |
- Most of the saving is the effort level. The gap between effort high only and all three changes is inside the run-to-run variation.
- On the task built for the other two changes (read-only subagents): baseline $3.95, effort high only $2.88, Widal.Code $1.10, one session each.
Results by task category
Δ cost against the baseline, side-by-side run of 21 November 2026.
| Category | Tasks | Baseline | Widal.Code | caveman | Headroom | RTK |
|---|---|---|---|---|---|---|
| Bug fixes | 3 | $3.17 | −52% | −31% | −28% | −18% |
| Features | 2 | $5.67 | −70% | −59% | −24% | −28% |
| Investigation, refactoring and documentation | 3 | $3.06 | −39% | −23% | −13% | −9% |
| Hard tasks | 3 | $9.54 | −56% | −40% | −19% | 0% |
| Tasks with a known waste built in | 2 | $5.35 | −69% | −12% | −15% | +2% |
- Shading grows with the cut in cost. Each cell rests on two or three sessions.
Per-task results
Each row is one task. Pick a setup to compare with the baseline.
- Baseline
- Widal.Code
- Bug fix, easy: $0.33 on the baseline, $0.32 with Widal.Code, 2% lower.
- Bug fix, medium: $1.33 on the baseline, $0.50 with Widal.Code, 62% lower.
- Bug fix, many rules: $1.52 on the baseline, $0.70 with Widal.Code, 54% lower.
- Feature, medium: $3.40 on the baseline, $1.27 with Widal.Code, 63% lower.
- Feature, harder: $2.26 on the baseline, $0.45 with Widal.Code, 80% lower.
- Investigation: $1.26 on the baseline, $0.79 with Widal.Code, 38% lower.
- Refactoring: $1.56 on the baseline, $0.85 with Widal.Code, 46% lower.
- Documentation: $0.24 on the baseline, $0.23 with Widal.Code, 2% lower.
- Query language: $4.38 on the baseline, $1.77 with Widal.Code, 60% lower.
- Week numbers: $1.67 on the baseline, $0.84 with Widal.Code, 50% lower.
- Speed problem: $3.49 on the baseline, $1.56 with Widal.Code, 55% lower.
- Read-only subagents: $4.04 on the baseline, $1.12 with Widal.Code, 72% lower.
- Long command output: $1.31 on the baseline, $0.56 with Widal.Code, 58% lower.
Point at a task or tap it for its numbers. The arrow keys move through the tasks, and Enter keeps one open.
All 13 tasks
Widal.Code 0.3.1 against the baseline, 21 November 2026. Cost at list prices.
59% lower
- Baseline
- $26.79
- Widal.Code
- $10.97
13 of 13 tasks cost less with Widal.Code. All 13 were finished in both setups.
Show these numbers as a table
| Task | Hidden checks | Baseline | Widal.Code | Difference |
|---|---|---|---|---|
| Bug fix, easy | 10 | $0.33 | $0.32 | 2% lower |
| Bug fix, medium | 10 | $1.33 | $0.50 | 62% lower |
| Bug fix, many rules | 14 | $1.52 | $0.70 | 54% lower |
| Feature, medium | 8 | $3.40 | $1.27 | 63% lower |
| Feature, harder | 15 | $2.26 | $0.45 | 80% lower |
| Investigation | 9 | $1.26 | $0.79 | 38% lower |
| Refactoring | 8 | $1.56 | $0.85 | 46% lower |
| Documentation | 7 | $0.24 | $0.23 | 2% lower |
| Query language | 29 | $4.38 | $1.77 | 60% lower |
| Week numbers | 14 | $1.67 | $0.84 | 50% lower |
| Speed problem | 8 | $3.49 | $1.56 | 55% lower |
| Read-only subagents | 6 | $4.04 | $1.12 | 72% lower |
| Long command output | 10 | $1.31 | $0.56 | 58% lower |
| All 13 tasks | 148 | $26.79 | $10.97 | 59% lower |
Effort levels
Cost of the 13 tasks per model and effort level, no tool, each model's levels side by side on 21 Nov 2026. Every cell solved 13/13.
| Model | Low | Medium | High | Xhigh | Max | Widal.Code |
|---|---|---|---|---|---|---|
| Haiku 5.5 | $0.11−75% | $0.16−65% | $0.23−47% | $0.44 | $2.285.1× | not run |
| Sonnet 5.5 | $1.21−68% | $1.33−65% | $1.98−48% | $3.78 | $17.394.6× | $1.87−51% |
| Opus 5.5 | $2.57−72% | $3.96−57% | $4.79−48% | $9.23 | $30.013.3× | $4.41−52% |
| Fable 5.2 | $7.03†−79% | $8.62−71% | $13.11−55% | $29.38 | $102.44†3.0× | $12.10−59% |
- Second line: against effort xhigh of the same model, as a change or as a multiple.
- † Fable 5.2 at effort low and max ran next to a second effort xhigh run ($34.21) and are compared with that one.
- Haiku 5.5 with no effort level set (Claude Code's default for it): $0.17.
Setups
Each tool ran as its own install sets it up, with telemetry off and its data kept in the benchmark folder.
| Setup | Version | Mechanism | Configuration | Activity in the run |
|---|---|---|---|---|
| Baseline | Claude Code 2.1.330 | None | Fable 5.2, effort xhigh, no tool | |
| Widal.Code | 0.3.1 | Reads the baseline's logs, proposes settings, applies them | Effort high, subagents on Sonnet 5.5, Bash output capped at 10,000 characters | 482 hook calls in its 39 sessions on Fable 5.2 |
| caveman | 4.1.0 | Instruction to answer in very short sentences | Plugin, CAVEMAN_DEFAULT_MODE=caveman (it is off in headless sessions otherwise) | Mode on in 13 of 13 sessions |
| Headroom | 0.37.0 | Local proxy that compresses each request | Own proxy and home folder, default mode cache, telemetry off | All requests proxied; its token counts equal Claude Code's |
| RTK | 1.2.1 | Hook that filters the output of shell commands | Apache-2.0 build (rtk-oss), hook and RTK.md as set up by rtk init -g, telemetry off | 148 commands rewritten, 2,100 tokens removed |
"model": "claude-fable-5-2",Removed: - "effortLevel": "xhigh",Added: + "effortLevel": "high",Added: + "bashOutputMaxChars": 10000,Added: + "env": { "CLAUDE_CODE_SUBAGENT_MODEL": "claude-sonnet-5-5" }
Tasks
One sample project: a command line tool in JavaScript with no packages. Tasks, hidden checks and reference solutions are in the repository.
| ID | Task | Category | Hidden checks |
|---|---|---|---|
| t1 | Bug fix, easyA list shows 9 rows on a page instead of 10, and the last page is missing. | Bug fixes | 10 |
| t2 | Bug fix, mediumThe reader for CSV files breaks on fields in quotes. | Bug fixes | 10 |
| t3 | Bug fix, many rulesAmounts of money are read and printed wrongly. The fix has to follow a long list of rules. | Bug fixes | 14 |
| t4 | Feature, mediumA flag that prints a report as JSON, for other programs to read. | Features | 8 |
| t5 | Feature, harderA cache with a size limit and an expiry time, written from a stub. | Features | 15 |
| t6 | InvestigationTwo reports are wrong, and the issue does not name the cause. | Investigation, refactoring and documentation | 9 |
| t7 | RefactoringOne function is split into three, and what it prints must not change. | Investigation, refactoring and documentation | 8 |
| t8 | DocumentationA section of the README is brought in line with the help text. | Investigation, refactoring and documentation | 7 |
| h1 | Query languageHard. A new command with a small query language and a long specification. | Hard tasks | 29 |
| h2 | Week numbersHard. Totals per ISO 8601 week, a rule that the issue only names. | Hard tasks | 14 |
| h3 | Speed problemHard. Every command has to be fast on a file of 200,000 rows, and the slowness has two causes. | Hard tasks | 8 |
| s1 | Read-only subagentsThree questions about the code, each given to a subagent that only reads. | Tasks with a known waste built in | 6 |
| s2 | Long command outputThe CSV bug fix again, in a project whose test command prints 75,000 characters first. | Tasks with a known waste built in | 10 |
Method
- Harness
- Claude Code 2.1.330 in headless mode (claude -p). One session per task and setup, no person present. The prompt is the issue text.
- Isolation
- A fresh copy of the project in a new git repository per session. No user settings, plugins, MCP servers or memory. The same short list of allowed tools, a budget and a time limit for every setup.
- Pairing
- In a side-by-side run every setup starts a task at the same moment, and the next task starts when all have finished. Times are compared only within one run.
- Scoring
- The task's hidden checks and the project's existing tests, run by the same scorer for every setup. A task is solved when both pass. No human grading.
- Cost
- Reported by Claude Code at Anthropic list prices, subagents included, and checked against Widal.Code's own computation from the logs.
- Statistics
- Totals over the 13 tasks. p is a two-sided sign test on the per-task differences against the baseline of the same run.
- Variation
- The baseline's total was $25.09, $29.38, $26.79 and $34.21 in four runs (36% apart). A difference under about 35% from one run is not conclusive.
- Exclusions
- One session excluded after a task setup error, and rerun. No session reached the time limit or ended with an error.
Validity checks
| Threat | Control | Result |
|---|---|---|
| Faulty task or check | A repository test proves each task is open at the start and solved by its reference solution | 13 of 13 tasks; 519 of 519 repository tests pass (22 Nov 2026) |
| Copying the reference solution | Solutions and checks are outside the session folder; the prompt is the issue only | Median 2% of added long lines match the reference, at most 30% (385 sessions without other tools) |
| Reading the hidden checks | Not blocked | Not measured |
| User configuration leaking in | Sessions start with project settings only and an empty MCP server list | Set by the session's start options |
| A tool that did not act | Each tool's own counter is read; one warm-up task per tool, not counted | Every tool acted in its sessions (see Setups) |
| Wrong cost | Two separate computations of every session's cost | Equal in 523 of 524 sessions; one differs by $0.09 (under 1%) |
| Uneven machine load | Side-by-side runs | Times of the first run, which was not side by side, are excluded |
| Chance | Sign test; Widal.Code and the baseline ran three times on Fable 5.2 | 38 of 39 tasks cheaper with Widal.Code, p <0.001 |
| Tasks built for Widal.Code | Result recomputed without the two tasks with waste built in | −55% without them, over three runs |
| Version drift | Versions recorded in every session | All 21 Nov sessions: Claude Code 2.1.330, Widal.Code 0.3.1, Node 24.11.1 |
| Unequal permissions | The same allowed tools for every setup | Refused tool calls: baseline 43, Widal.Code 23, caveman 23, Headroom 41, RTK 42 |
Limitations
- 13 tasks in one small JavaScript project. That is too few to separate setups that differ by little: telling a 10% cost difference from noise takes about 100 tasks.
- Every setup solved every task, so the benchmark shows nothing about differences in quality.
- caveman, Headroom and RTK ran once each. The results of Headroom (−19%) and RTK (−9%) are inside the run-to-run variation, and these tasks have little of the long command output that both are built for.
- One computer, one account, two days. The tasks and solutions are public and were written with a model of the same family, so later models may have seen them (source 1).
- Sessions could read files outside their folder, including the hidden checks, and the logs were not analysed for it (source 2).
- Savings are measured from effort xhigh. A team that already works at effort high saves less.
Reproduce
A run starts real Claude Code sessions with your own sign-in and counts against your plan. Baseline and Widal.Code on the 13 tasks: 26 sessions, about $40 at list prices.
git clone https://github.com/Widal-Inc/widal-code && cd widal-code node bench/run.mjs --out <folder> --arms fable-xhigh node bench/widal-on.mjs --out <folder> node bench/run.mjs --out <folder> --round 2 --paired --arms fable-xhigh,widal-on node bench/widal-on.mjs --out <folder> --undo node bench/report.mjs --out <folder>
Other tools run from an arms file of your own with --arms-file. Full instructions: bench/README.md.
Changelog
- 22 Nov 2026
- Leaderboard layout with 95% intervals. caveman, Headroom and RTK shown. Times kept to 0.1 s, which moves caveman's time from −48% to −47%.
- 21 Nov 2026
- Effort levels of four models, Widal.Code on Sonnet 5.5 and Opus 5.5, and the side-by-side run with three other tools. Wall times of the first run withdrawn.
- 20 Nov 2026
- First run: baseline and Widal.Code on 13 tasks.
Sources
- OpenAI, "Why we no longer evaluate SWE-bench Verified", 23 Feb 2026
- Kirgis et al., "Log analysis is necessary for credible evaluation of AI agents", arXiv 2605.08545, 8 May 2026
Compiled from the records of 523 sessions run on 20 and 21 Nov 2026 and the code in github.com/Widal-Inc/widal-code. Model names and prices are Anthropic's. Tool descriptions are from each tool's README, read on 21 Nov 2026.