Editorial illustration for Google AI launches Auto-Diagnose, LLM tool flags 84.3% of reports as ‘Please fix’
Google AI Auto-Diagnose Flags 84% of Dev Test Errors
Google AI launches Auto-Diagnose, LLM tool flags 84.3% of reports as ‘Please fix’
Auto-Diagnose is not gentle. Of 517 feedback reports, 436 came back with a blunt directive: “Please fix.” That’s 84.3%, an overwhelming majority. The tool isn’t polite.
It’s precise. And developers seem to want it that way. Across 370 reviewers, the “Not helpful” rate sits at just 5.8%, well below Google’s kill threshold.
When it does fail, those failures reveal hidden cracks: missing crash logs, unrecorded component states. Real infrastructure bugs, surfaced as a side effect. Auto-Diagnose ranks 14th in helpfulness among 370 tools, top 3.78%.
It hits 90.14% root-cause accuracy on real-world integration test failures. This is an AI that doesn’t just diagnose. It demands action.
A team of Google researchers introduced Auto-Diagnose, an LLM-powered tool that automatically reads the failure logs from a broken integration test, finds the root cause, and posts a concise diagnosis directly into the code review where the failure showed up.
The numbers tell the story, but the real narrative is in what they reveal. Auto-Diagnose isn’t just flagging failures, it’s forcing action. 84.3% of reports demand fixes.
That’s not noise; it’s a signal that reviewers trust the diagnosis enough to act. And when a tool’s “not helpful” rate sits below 5.8%, well under Google’s kill-switch threshold, you’re looking at a system that earns its keep. But the quiet win might be the one nobody planned for.
Seven manual failures turned into four missing crash logs and three missing component logs, real infrastructure bugs, now patched. Another twenty “more information needed” cases surfaced similar issues in production. Auto-Diagnose isn’t just diagnosing tests; it’s diagnosing the diagnostic pipeline itself.
90.14% root-cause accuracy on real-world failures across 39 teams. A rank of 14th out of 370 tools, top 3.78%. And a problem that 6,059 developers called one of their top five headaches.
The tool works. The question now isn’t whether to use it, it’s how many more hidden cracks in the foundation it will expose along the way.
Related Reading
Browse all Google news →
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
Related Topic NVIDIA and Google Cloud let developers scale AI from prototype to production
Market Trends
Related Topic Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks
Same Week 91% of businesses now use video marketing — AI cut the cost of keeping up by 91% too
Market Trends
From Archives Penligent and Giskard among top AI red‑team tools for model security
From Archives AI protein-design tools offer flexible workflows for any protein class
From Archives Google's AI mode will open linked pages beside search on Chrome desktop
From Archives Google adds ‘Skills’ to Chrome, enabling one‑click Gemini AI prompts
Business & Startups
Common Questions Answered
How accurate is Google's Auto-Diagnose system in identifying integration-test failures?
Google's Auto-Diagnose system demonstrated high accuracy in flagging integration-test failures, with 84.3% of feedback reports receiving a 'Please fix' response from reviewers. In a manual evaluation of 71 real-world failures, the system successfully produced a diagnosis for each case, indicating its potential effectiveness in automated bug triage.
What percentage of developers found the Auto-Diagnose tool helpful?
According to the study, developers rated the Auto-Diagnose tool helpful in approximately 62.96% of interactions. The tool also maintained a low 'Not helpful' rate of 5.8%, which is well below Google's 10% threshold for maintaining a tool's viability.
How many developers and feedback reports were involved in the Auto-Diagnose trial?
The Auto-Diagnose system was trialed with 437 distinct developers who generated 517 feedback reports. Of these reports, 370 reviewers classified the diagnoses, with 436 reports (84.3%) receiving a 'Please fix' response, demonstrating significant engagement with the tool's suggestions.
