Editorial illustration for SWE-agent: coding agents solve tasks, but setup affects wasted work
Coding Agents Pass Tests But Leave Messy Code Behind
SWE-agent: coding agents solve tasks, but setup affects wasted work
A coding agent can pass every test and still leave a mess behind. That's the finding buried in a new set of experiments on SWE-agent-style workflows: 50,400 runs total, split across a 24,000-run compliance sweep and 14,400 runs testing individual harness rules, built to answer one question. When an agent takes extra turns to reach a working patch, how much of that extra work actually mattered, and does anything in the surrounding setup cut it down?
The runs use real repositories, real diffs, and real test failures, but the agent's decisions come from a seeded Python policy rather than Claude Code, Codex, or any live model. That tradeoff is deliberate. It means every turn, every patch, every wasted edit can be traced and counted instead of guessed at.
Standard pass/fail scoring can't tell a clean five-turn fix from an eleven-turn run that padded its final patch with changes nobody needed. Both get marked solved. Neither score says anything about the extra lines touched, the files edited for no reason, or the churn that crept into the final diff along the way.
For sloppy agents, adaptive leaves a final patch that is 0.97 ± 0.35 lines smaller on average. For loop-prone agents, the difference is 0.71 ± 0.29 lines. But total churn goes up by 0.66 ± 0.54 and 1.00 ± 0.45 lines respectively.
Why this matters
The 50,400 runs make a simple point that teams building on coding agents keep skipping: a passing test doesn't mean a clean patch. Six unnecessary lines out of eight in a "successful" run isn't a rounding error, it's wasted diff that someone has to review, maintain, or quietly ship into production. For developers wiring these agents into CI, "task solved" is the wrong metric to optimize alone.
For founders selling agent-based tooling, this is a warning that benchmarks measuring pass/fail rates are hiding real cost, the kind that shows up later as code review overhead or regressions nobody traced back to turn 7 of an agent run. For researchers, the finding that harness rules, how an agent edits, navigates, and tests, change outcomes independent of the model itself is worth taking seriously. It means some of what gets credited to model capability is actually scaffolding design.
We'd treat "turns past first success" and "lines changed versus reference fix" as metrics worth logging by default, not afterthoughts. Cheap to track, and apparently very informative about what these agents are actually doing when nobody's grading the diff.
Related Reading
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
Open Source Box CEO Aaron Levie Says AI Will Drive "Biggest Cybersecurity Upgrade
Open Source Study: 6.7% of Deepfake Requests on Hugging Face Targeted Children
From Archives Epic shifts Disney shooter to AI tools like Claude, Bloomberg says
From Archives Open Models Offer Benefits, But an Extractive Approach Raises Safety Risks
Common Questions Answered
What did the SWE-agent experiments reveal about passing tests and code quality?
The experiments across 50,400 runs demonstrated that a coding agent can pass every test while still leaving unnecessary code behind. The research found that successful patches often contained significant wasted diff that developers would need to review and maintain, proving that passing tests alone don't guarantee clean or efficient code.
How does adaptive setup affect patch size for sloppy versus loop-prone agents?
For sloppy agents, adaptive setup produces final patches that are 0.97 ± 0.35 lines smaller on average, while for loop-prone agents the difference is 0.71 ± 0.29 lines. However, total churn increases by 0.66 ± 0.54 and 1.00 ± 0.45 lines respectively, indicating that smaller patches come at the cost of more intermediate work.
Why is optimizing solely for 'task solved' problematic when using coding agents in CI?
Optimizing only for task completion ignores the wasted diff and unnecessary lines that agents introduce during their problem-solving process. Developers must review and maintain this extra code, and the accumulated technical debt from shipping inefficient patches into production can become a significant burden for teams.
What does the research suggest about current benchmarks for agent-based coding tools?
The findings indicate that existing benchmarks measuring only whether agents solve tasks are insufficient for evaluating coding agent performance. The research warns that founders and developers need to consider metrics beyond task completion, including patch efficiency and code quality, to properly assess agent-based tooling.
Further Reading
- Does Code Cleanliness Affect Coding Agents? A Controlled Study - arXiv
- Do Coding Agents Reuse Existing Code or Reinvent the Wheel? - arXiv
- When Does Restricting a Coding Agent to execute_code Improve Software Engineering? - arXiv
- A Benchmark for Evaluating Software Development Agents - arXiv
- SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering - arXiv Science
