Editorial illustration for Study: Fancy coding harnesses cost more but not better across 447 tasks
Fancy Coding Tools Don't Improve Performance, Study Finds
Study: Fancy coding harnesses cost more but not better across 447 tasks
Two researchers at Shandong University ran 447 coding tasks through five language models and three agent harnesses, then checked whether the fancier tooling actually earned its keep. The answer, laid out in a paper titled "What does a harness buy? Mostly tokens," is no.
Claude Code and OpenCode come loaded with specialized file and search tools, long system prompts, and automatic history compression. Mini-SWE-agent strips all that away, giving the model nothing but a Bash tool. Swap one harness for another, the study finds, and scores barely move, often less than the variation you get just rerunning the identical setup twice.
The costs don't follow that pattern. Running the heavier harnesses can triple token spend without a matching jump in task completion. That matters because leaderboards right now mix and match harnesses freely, and vendors lean on small score gaps to market their own wrapper around a given model.
Liu and Han, the paper's authors, set out to test whether those gaps reflect real engineering gains or just noise baked into how benchmarks get run.
Makers tout score gains from new agent harnesses like Claude Code or OpenCode. A study finds that claim doesn't hold up. Swapping harnesses barely moves results more than running the same setup twice, but costs can triple.
Why this matters
For teams picking a coding agent, this is a useful gut check against marketing. Harness vendors have built a whole genre of benchmark posts implying their scaffolding is what makes the difference between a model that fumbles a repo and one that ships clean patches. The 447-task comparison says otherwise: Claude Code and mini-SWE-agent land within two points of each other, and the hardest 45 tasks, where a fancier harness should theoretically earn its keep, show no such payoff.
What does move is cost, sometimes tripling for the same output. We'd read this as a prompt to stop treating harness choice as a performance lever and start treating it as an infrastructure decision, weighted by latency, tooling fit, and price per task rather than leaderboard bragging rights. If you're a founder budgeting API spend or a researcher designing the next eval, the model underneath still looks like the thing worth optimizing.
The scaffolding around it is mostly cosmetic, and paying triple for it buys little beyond nicer logs.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Kyutai Trains 1.2B-Parameter Model on 76,000 Hours of Video
From Archives OpenAI to stop training AI models on live internet over access risks
Common Questions Answered
What did the Shandong University study find about fancy coding harnesses like Claude Code and OpenCode?
The study tested 447 coding tasks across five language models and three agent harnesses and found that fancy harnesses with specialized tools, long system prompts, and automatic history compression did not produce better results than simpler alternatives. Claude Code and mini-SWE-agent, despite their vastly different feature sets, landed within two points of each other in performance, suggesting that the added complexity and cost of fancy harnesses does not translate to meaningful improvements.
How much more expensive are fancy coding harnesses compared to simpler alternatives according to the research?
According to the study, costs can triple when using fancy harnesses like Claude Code and OpenCode compared to simpler setups like mini-SWE-agent. Despite this significant cost increase, the performance gains are minimal, making the premium pricing difficult to justify based on actual task completion results.
What is the difference between mini-SWE-agent and Claude Code in terms of tooling?
Mini-SWE-agent strips away specialized features and provides the model with nothing but a Bash tool, while Claude Code comes loaded with specialized file and search tools, long system prompts, and automatic history compression. Despite these substantial differences in complexity and features, the study found that both harnesses performed comparably on the 447 coding tasks tested.
Did fancy harnesses show any advantage on the hardest coding tasks in the study?
No, the study found that even on the hardest 45 tasks where a fancier harness should theoretically earn its keep, there was no performance payoff. This suggests that the advanced scaffolding in fancy harnesses does not provide the expected benefits even when tackling the most challenging coding problems.
Why is this study important for teams evaluating coding agents?
The study provides a useful reality check against marketing claims from harness vendors who often imply that their specialized scaffolding is what makes the difference between a model that fails and one that succeeds. By demonstrating that fancy harnesses don't deliver proportional performance improvements despite their higher costs, teams can make more informed decisions about which coding agents to adopt based on actual performance data rather than vendor marketing.
Further Reading
- HarnessTax: How Much Does the Harness Matter for Coding Agents? - Arena
- The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable - arXiv
- Benchmarking Coding Agents on Databricks' Multi-Million Line Codebase - Databricks
- Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering - arXiv
- Prompt-Induced Waste in Large Reasoning Models: A Preregistered Two-Harness Benchmark of Coding Agents - arXiv
