Editorial illustration for Study: AI coding agents locate correct file but miss key lines in bugs
Study: AI coding agents locate correct file but miss key...
AI coding assistants are great at finding the file. They're terrible at reading it. A new study shows these tools can successfully locate the correct source file a bug lives in, but when asked to pinpoint the exact problematic lines, their performance crumbles to near uselessness.
The research reveals a stubborn ceiling. General-purpose coding agents identify only 14 to 19 percent of the lines actually required to fix an issue. This isn't a problem of raw intelligence.
Throwing more powerful language models at the task does nothing to lift the score. GPT models lead the pack, but the pattern remains locked. Claude Code, Codex, OpenHands, and others all post nearly identical, mediocre results.
The architecture of the agent is irrelevant. The failure is consistent.
A new benchmark separates code search from the actual fix and exposes a hidden weakness of AI coding agents.
Specialized tools highlight the trade-off without solving it. AutoCodeRover is precise but conservative. OrcaLoca generates little noise but misses too much.
Neither bridges the fundamental gap. The study confirms this gap is structural. In experiments, the repair process completely fails if the AI's view of the relevant code context drops below a certain threshold.
It needs to see enough of the right lines, not just the right file name.
This creates an awkward workflow. The AI can take you to the correct room in the mansion. It then hands you a magnifying glass and expects you to find the hairline crack in the plaster yourself.
The tools have learned to navigate a codebase's geography. They recognize files and their relationships. They cannot yet parse the specific logic flaw, the single integer overflow, the one-off error hiding in plain sight.
That distinction matters. It means these agents are useful guides, but they remain assistants, not engineers. They find the chapter.
You read the page.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives OpenAI confirms cooperation as state attorneys general launch investigation
From Archives Gemini‑SQL2 leads BIRD benchmark with 80.04% execution accuracy
Common Questions Answered
Why do AI coding agents fail at identifying specific bug lines even when they find the correct file?
AI coding agents struggle with pinpointing exact problematic lines because they only identify 14 to 19 percent of the lines actually required to fix an issue, despite successfully locating the correct source file. This limitation is not due to raw intelligence but rather a structural gap in how these tools process and analyze code context within files.
What does the study reveal about the performance threshold for AI code repair processes?
The research confirms that AI repair processes completely fail if the AI's view of relevant code context drops below a certain threshold, meaning the tool needs to see enough of the right lines, not just identify the right file name. This structural limitation demonstrates that context visibility is critical to successful bug fixing.
How do specialized tools like AutoCodeRover and OrcaLoca address the bug-fixing limitations?
AutoCodeRover is precise but conservative in its approach, while OrcaLoca generates little noise but misses too much relevant code. However, neither specialized tool successfully bridges the fundamental gap between locating files and identifying the specific lines needed for repairs.
What workflow challenge does the gap between file location and line identification create for users?
The structural limitation creates an awkward workflow where AI can take users to the correct file but cannot reliably pinpoint the exact lines requiring fixes. This forces developers to manually review and identify the problematic code sections despite the AI's initial file-finding capability.
Further Reading
- Are bugs and incidents inevitable with AI coding agents? — Stack Overflow Blog
- How Coding Agents Fail Their Users: A Large-Scale Analysis ... — arXiv
- 9 Critical Failure Patterns of Coding Agents — DAPLab
- 5 Facts About AI Coding Agents from Comprehensive Benchmarking — DevOps.com
- Hard Won Lessons from Building Effective AI Coding Agents — YouTube
