How the comparison was run
Arena researchers evaluated 21 combinations of seven models and three coding-agent harnesses: Claude Code, Codex CLI and the minimal open-source Pi harness. Each pair was run three times on the same 30 randomly sampled SWE-bench Lite tasks and 30 Terminal-Bench 2.0 tasks. The study used each harness's native high-effort configuration, capped attempts at 100 agent turns and scored results with the benchmarks' official evaluators.
Costs were calculated from a fixed direct-API price list dated 1 September 2026, with the same model price applied across harnesses. The researchers averaged the three attempts for each task and used 10,000 bootstrap resamples for 95% confidence intervals. External network access was blocked for SWE-bench Lite and the default web tools in Claude Code and Codex were disabled for that benchmark.
The reported gap is mostly in spending
On these tasks, Arena reports that changing the harness moved average success rates by only about plus or minus two percentage points on SWE-bench Lite and about plus or minus five points on Terminal-Bench 2.0, while costs could diverge much more. Claude Fable 5, for example, solved 97.8% of SWE-bench Lite attempts in Claude Code and 96.7% in Pi, while the reported average cost was $1.33 versus $0.67.
Across models shared by the harnesses, the authors calculate that Claude Code cost about twice as much as Pi and 1.6 times as much as Codex on SWE-bench Lite, using geometric means of cost ratios. On Terminal-Bench 2.0, Claude Code cost about 1.5 times as much as Pi. Pi reached the study's cost-success Pareto frontier on both benchmarks with only four tools: read, write, edit and bash.
Provider-native harnesses did not always lead
The study also found that a model's provider-specific harness was not consistently its highest-scoring pairing. Across six Anthropic and OpenAI models and both benchmarks, an alternative harness had the highest observed success rate in nine of twelve comparisons. GPT-5.6 Sol, for example, reached 83.3% success in Pi on the sampled Terminal-Bench tasks versus 78.9% in Codex, at a reported $0.42 versus $0.76 average cost.
The authors point to initial context as one possible contributor to cost. Across all seven models, Claude Code's mean initial context was more than ten times Pi's in their traces. That observation does not by itself explain total cost, which also depends on caching, output length and later calls, but it makes harness prompt and tool-schema overhead measurable rather than invisible.
Limits come before generalising the result
HarnessTax is an author-run benchmark study published by Arena, not a peer-reviewed paper or an independent audit of the three products. It covers only two public benchmarks and 30 sampled tasks from each. The authors explicitly note that the evaluated models may have encountered those benchmarks during training and that richer harness features could matter on other workloads, especially long-running or interactive development tasks.
The Hacker News discussion that surfaced the work is useful discovery context but does not establish the benchmark claims. The results therefore support a narrower conclusion: on this particular controlled comparison, harness choice changed measured cost much more than average task success. Reproduction on private repositories, longer sessions and different task distributions will be needed before treating the reported cost ratios as typical of everyday coding work.