Version 0.1 · updated 22 Nov 2026

Widal.Code Benchmark

Cost, time and solve rate of Claude Code setups on real agent sessions. Each setup runs the same tasks, is scored by hidden tests, and is compared with the same model running with no tool.

Tasks
13
Hidden checks
148
Sessions
523
Baseline
Fable 5.2, xhigh
Harness
Claude Code 2.1.330

Leaderboard

Side-by-side run, 21 November 2026. Every setup on Fable 5.2 at effort xhigh. Sorted by cost.

#SetupSolvedCostΔ cost95% CICheaper taskspWall timeΔ timeOutput tokensTurns
1Widal.Code 0.3.113/13$10.97−59%−65% to −52%13/13<0.00140 min−43%150k162
2caveman 4.1.013/13$17.27−36%−47% to −22%11/130.02237 min−47%226k190
3Headroom 0.37.013/13$21.58−19%−31% to −9%11/130.02253 min−25%325k286
4RTK 1.2.113/13$24.48−9%−22% to +3%9/130.26765 min−7%361k283
Baseline no tool13/13$26.7970 min395k251
  • Solved: every hidden check passes and the project's existing tests still pass. All setups solved 148 of 148 checks.
  • Cost: Claude Code's reported cost at Anthropic list prices, subagents included. Wall time: start to end of each session, summed.
  • Δ against the baseline of the same run. 95% CI: interval of Δ cost from resampling the 13 tasks (10,000 draws). It covers the choice of tasks, not run-to-run variation. Cheaper tasks and p: per-task cost against the baseline, two-sided sign test.
  • The baseline varied by 36% between four runs. Single-run differences under about 35% (Headroom, RTK) are not conclusive.

Widal.Code against baseline

Per model, from effort xhigh with no tool. Widal.Code's changes are the ones it proposed from the baseline's own sessions.

ModelRuns per taskBaselineWidal.CodeΔ cost95% CICheaper taskspΔ timeSolved
Fable 5.23$81.26$34.64−57%−61% to −53%38/39<0.001−47%*39/39
Sonnet 5.51$3.78$1.87−51%−58% to −41%13/13<0.001−58%13/13
Opus 5.51$9.23$4.41−52%−57% to −47%13/13<0.001−44%13/13
  • * Two side-by-side runs. The first Fable 5.2 run was not side by side and its times are excluded.
  • Changes applied: effort high on all three models, subagents on Sonnet 5.5 on Fable 5.2 and Opus 5.5, Bash output capped at 10,000 characters on Fable 5.2. Not run on Haiku 5.5.

Ablation

Fable 5.2, one side-by-side run on 21 Nov 2026. Each effort change alone, against Widal.Code's three changes.

SetupCostΔ costSolved
Baseline, effort xhigh$29.3813/13
Effort high only$13.11−55%13/13
Widal.Code, all three changes$12.10−59%13/13
Effort medium only$8.62−71%13/13
  • Most of the saving is the effort level. The gap between effort high only and all three changes is inside the run-to-run variation.
  • On the task built for the other two changes (read-only subagents): baseline $3.95, effort high only $2.88, Widal.Code $1.10, one session each.

Results by task category

Δ cost against the baseline, side-by-side run of 21 November 2026.

CategoryTasksBaselineWidal.CodecavemanHeadroomRTK
Bug fixes3$3.17−52%−31%−28%−18%
Features2$5.67−70%−59%−24%−28%
Investigation, refactoring and documentation3$3.06−39%−23%−13%−9%
Hard tasks3$9.54−56%−40%−19%0%
Tasks with a known waste built in2$5.35−69%−12%−15%+2%
  • Shading grows with the cut in cost. Each cell rests on two or three sessions.

Per-task results

Each row is one task. Pick a setup to compare with the baseline.

  • Baseline
  • Widal.Code
  1. Bug fix, easy: $0.33 on the baseline, $0.32 with Widal.Code, 2% lower.
  2. Bug fix, medium: $1.33 on the baseline, $0.50 with Widal.Code, 62% lower.
  3. Bug fix, many rules: $1.52 on the baseline, $0.70 with Widal.Code, 54% lower.
  4. Feature, medium: $3.40 on the baseline, $1.27 with Widal.Code, 63% lower.
  5. Feature, harder: $2.26 on the baseline, $0.45 with Widal.Code, 80% lower.
  6. Investigation: $1.26 on the baseline, $0.79 with Widal.Code, 38% lower.
  7. Refactoring: $1.56 on the baseline, $0.85 with Widal.Code, 46% lower.
  8. Documentation: $0.24 on the baseline, $0.23 with Widal.Code, 2% lower.
  9. Query language: $4.38 on the baseline, $1.77 with Widal.Code, 60% lower.
  10. Week numbers: $1.67 on the baseline, $0.84 with Widal.Code, 50% lower.
  11. Speed problem: $3.49 on the baseline, $1.56 with Widal.Code, 55% lower.
  12. Read-only subagents: $4.04 on the baseline, $1.12 with Widal.Code, 72% lower.
  13. Long command output: $1.31 on the baseline, $0.56 with Widal.Code, 58% lower.

Point at a task or tap it for its numbers. The arrow keys move through the tasks, and Enter keeps one open.

All 13 tasks

Widal.Code 0.3.1 against the baseline, 21 November 2026. Cost at list prices.

59% lower

Baseline
$26.79
Widal.Code
$10.97

13 of 13 tasks cost less with Widal.Code. All 13 were finished in both setups.

Show these numbers as a table
Cost of each task on the baseline and with Widal.Code, side by side on 21 November 2026
TaskHidden checksBaselineWidal.CodeDifference
Bug fix, easy10$0.33$0.322% lower
Bug fix, medium10$1.33$0.5062% lower
Bug fix, many rules14$1.52$0.7054% lower
Feature, medium8$3.40$1.2763% lower
Feature, harder15$2.26$0.4580% lower
Investigation9$1.26$0.7938% lower
Refactoring8$1.56$0.8546% lower
Documentation7$0.24$0.232% lower
Query language29$4.38$1.7760% lower
Week numbers14$1.67$0.8450% lower
Speed problem8$3.49$1.5655% lower
Read-only subagents6$4.04$1.1272% lower
Long command output10$1.31$0.5658% lower
All 13 tasks148$26.79$10.9759% lower

Effort levels

Cost of the 13 tasks per model and effort level, no tool, each model's levels side by side on 21 Nov 2026. Every cell solved 13/13.

ModelLowMediumHighXhighMaxWidal.Code
Haiku 5.5$0.11−75%$0.16−65%$0.23−47%$0.44$2.285.1×not run
Sonnet 5.5$1.21−68%$1.33−65%$1.98−48%$3.78$17.394.6×$1.87−51%
Opus 5.5$2.57−72%$3.96−57%$4.79−48%$9.23$30.013.3×$4.41−52%
Fable 5.2$7.03†−79%$8.62−71%$13.11−55%$29.38$102.44†3.0×$12.10−59%
  • Second line: against effort xhigh of the same model, as a change or as a multiple.
  • † Fable 5.2 at effort low and max ran next to a second effort xhigh run ($34.21) and are compared with that one.
  • Haiku 5.5 with no effort level set (Claude Code's default for it): $0.17.

Setups

Each tool ran as its own install sets it up, with telemetry off and its data kept in the benchmark folder.

SetupVersionMechanismConfigurationActivity in the run
BaselineClaude Code 2.1.330NoneFable 5.2, effort xhigh, no tool
Widal.Code0.3.1Reads the baseline's logs, proposes settings, applies themEffort high, subagents on Sonnet 5.5, Bash output capped at 10,000 characters482 hook calls in its 39 sessions on Fable 5.2
caveman4.1.0Instruction to answer in very short sentencesPlugin, CAVEMAN_DEFAULT_MODE=caveman (it is off in headless sessions otherwise)Mode on in 13 of 13 sessions
Headroom0.37.0Local proxy that compresses each requestOwn proxy and home folder, default mode cache, telemetry offAll requests proxied; its token counts equal Claude Code's
RTK1.2.1Hook that filters the output of shell commandsApache-2.0 build (rtk-oss), hook and RTK.md as set up by rtk init -g, telemetry off148 commands rewritten, 2,100 tokens removed
Widal.Code's settings change on Fable 5.2
"model": "claude-fable-5-2",Removed: - "effortLevel": "xhigh",Added: + "effortLevel": "high",Added: + "bashOutputMaxChars": 10000,Added: + "env": { "CLAUDE_CODE_SUBAGENT_MODEL": "claude-sonnet-5-5" }

Tasks

One sample project: a command line tool in JavaScript with no packages. Tasks, hidden checks and reference solutions are in the repository.

IDTaskCategoryHidden checks
t1Bug fix, easyA list shows 9 rows on a page instead of 10, and the last page is missing.Bug fixes10
t2Bug fix, mediumThe reader for CSV files breaks on fields in quotes.Bug fixes10
t3Bug fix, many rulesAmounts of money are read and printed wrongly. The fix has to follow a long list of rules.Bug fixes14
t4Feature, mediumA flag that prints a report as JSON, for other programs to read.Features8
t5Feature, harderA cache with a size limit and an expiry time, written from a stub.Features15
t6InvestigationTwo reports are wrong, and the issue does not name the cause.Investigation, refactoring and documentation9
t7RefactoringOne function is split into three, and what it prints must not change.Investigation, refactoring and documentation8
t8DocumentationA section of the README is brought in line with the help text.Investigation, refactoring and documentation7
h1Query languageHard. A new command with a small query language and a long specification.Hard tasks29
h2Week numbersHard. Totals per ISO 8601 week, a rule that the issue only names.Hard tasks14
h3Speed problemHard. Every command has to be fast on a file of 200,000 rows, and the slowness has two causes.Hard tasks8
s1Read-only subagentsThree questions about the code, each given to a subagent that only reads.Tasks with a known waste built in6
s2Long command outputThe CSV bug fix again, in a project whose test command prints 75,000 characters first.Tasks with a known waste built in10

Method

Harness
Claude Code 2.1.330 in headless mode (claude -p). One session per task and setup, no person present. The prompt is the issue text.
Isolation
A fresh copy of the project in a new git repository per session. No user settings, plugins, MCP servers or memory. The same short list of allowed tools, a budget and a time limit for every setup.
Pairing
In a side-by-side run every setup starts a task at the same moment, and the next task starts when all have finished. Times are compared only within one run.
Scoring
The task's hidden checks and the project's existing tests, run by the same scorer for every setup. A task is solved when both pass. No human grading.
Cost
Reported by Claude Code at Anthropic list prices, subagents included, and checked against Widal.Code's own computation from the logs.
Statistics
Totals over the 13 tasks. p is a two-sided sign test on the per-task differences against the baseline of the same run.
Variation
The baseline's total was $25.09, $29.38, $26.79 and $34.21 in four runs (36% apart). A difference under about 35% from one run is not conclusive.
Exclusions
One session excluded after a task setup error, and rerun. No session reached the time limit or ended with an error.

Validity checks

ThreatControlResult
Faulty task or checkA repository test proves each task is open at the start and solved by its reference solution13 of 13 tasks; 519 of 519 repository tests pass (22 Nov 2026)
Copying the reference solutionSolutions and checks are outside the session folder; the prompt is the issue onlyMedian 2% of added long lines match the reference, at most 30% (385 sessions without other tools)
Reading the hidden checksNot blockedNot measured
User configuration leaking inSessions start with project settings only and an empty MCP server listSet by the session's start options
A tool that did not actEach tool's own counter is read; one warm-up task per tool, not countedEvery tool acted in its sessions (see Setups)
Wrong costTwo separate computations of every session's costEqual in 523 of 524 sessions; one differs by $0.09 (under 1%)
Uneven machine loadSide-by-side runsTimes of the first run, which was not side by side, are excluded
ChanceSign test; Widal.Code and the baseline ran three times on Fable 5.238 of 39 tasks cheaper with Widal.Code, p <0.001
Tasks built for Widal.CodeResult recomputed without the two tasks with waste built in−55% without them, over three runs
Version driftVersions recorded in every sessionAll 21 Nov sessions: Claude Code 2.1.330, Widal.Code 0.3.1, Node 24.11.1
Unequal permissionsThe same allowed tools for every setupRefused tool calls: baseline 43, Widal.Code 23, caveman 23, Headroom 41, RTK 42

Limitations

  • 13 tasks in one small JavaScript project. That is too few to separate setups that differ by little: telling a 10% cost difference from noise takes about 100 tasks.
  • Every setup solved every task, so the benchmark shows nothing about differences in quality.
  • caveman, Headroom and RTK ran once each. The results of Headroom (−19%) and RTK (−9%) are inside the run-to-run variation, and these tasks have little of the long command output that both are built for.
  • One computer, one account, two days. The tasks and solutions are public and were written with a model of the same family, so later models may have seen them (source 1).
  • Sessions could read files outside their folder, including the hidden checks, and the logs were not analysed for it (source 2).
  • Savings are measured from effort xhigh. A team that already works at effort high saves less.

Reproduce

A run starts real Claude Code sessions with your own sign-in and counts against your plan. Baseline and Widal.Code on the 13 tasks: 26 sessions, about $40 at list prices.

git clone https://github.com/Widal-Inc/widal-code && cd widal-code
node bench/run.mjs --out <folder> --arms fable-xhigh
node bench/widal-on.mjs --out <folder>
node bench/run.mjs --out <folder> --round 2 --paired --arms fable-xhigh,widal-on
node bench/widal-on.mjs --out <folder> --undo
node bench/report.mjs --out <folder>

Other tools run from an arms file of your own with --arms-file. Full instructions: bench/README.md.

Changelog

22 Nov 2026
Leaderboard layout with 95% intervals. caveman, Headroom and RTK shown. Times kept to 0.1 s, which moves caveman's time from −48% to −47%.
21 Nov 2026
Effort levels of four models, Widal.Code on Sonnet 5.5 and Opus 5.5, and the side-by-side run with three other tools. Wall times of the first run withdrawn.
20 Nov 2026
First run: baseline and Widal.Code on 13 tasks.

Sources

  1. OpenAI, "Why we no longer evaluate SWE-bench Verified", 23 Feb 2026
  2. Kirgis et al., "Log analysis is necessary for credible evaluation of AI agents", arXiv 2605.08545, 8 May 2026

Compiled from the records of 523 sessions run on 20 and 21 Nov 2026 and the code in github.com/Widal-Inc/widal-code. Model names and prices are Anthropic's. Tool descriptions are from each tool's README, read on 21 Nov 2026.