Editorial illustration for AI coding agents need human cleanup on nearly 3 of 4 tasks in real-world test
AI Coding Agents Fail 73% of Real Tasks, Study Finds
AI coding agents need human cleanup on nearly 3 of 4 tasks in real-world test
GPT-6 Astra tops a new coding-agent leaderboard by hitting roughly 70 percent of a task's stated requirements. That sounds decent until you learn the same model gets the job fully right only 26.4 percent of the time. Researchers from Alibaba and several Asian universities built the test, called Co-Design Bench, specifically because older benchmarks stopped telling anyone anything useful. On those older tests, agents that can run their own code, check it, and patch their own mistakes routinely score above 99 percent, which makes it nearly impossible to tell one model from another.
Co-Design Bench ditches synthetic problems in favor of real change requests pulled from actual coding sessions, the kind where a user has to rephrase a request or roll back a change because the agent got it wrong the first time. Tasks range from small interface tweaks to bigger overhauls of existing web apps, sometimes with images, audio, video, or documents thrown in as part of the instructions. Ten models were tested. The pattern that shows up across most of them: the agent does the bulk of the work, then leaves one detail unresolved, leaving a human to finish the job.
According to the paper, GPT-6 Astra, the best of the ten models tested, meets about 70 percent of a task's requirements on average. But the result is fully correct in only 26.4 percent of tasks. In most cases the agent handles the bulk of the work but leaves at least one detail unfinished, so anyone using these tools almost always has to clean up by hand.
Why this matters
The gap between a 99 percent score on an old benchmark and a 25 percent completion rate on Co-Design Bench tells us what we already suspected: coding agents have been studying for a test with the answer key taped to the back. Letting a model run, check, and fix its own output measures whether it can satisfy a grader, not whether it can satisfy a user who wants an existing web app changed in a specific way, sometimes with a screenshot attached. For developers shipping agent-based tools, this is a reason to distrust marketing claims built on saturated benchmarks and to ask vendors which test their numbers come from.
For founders betting product roadmaps on "AI writes the app," a 75 percent human-cleanup rate on real change requests is a budget line, not a footnote. For researchers, Alibaba's team just raised the bar by grounding evaluation in actual user requests instead of synthetic tasks, which makes the field's progress claims harder to fake and easier to check. Watch whether other labs adopt Co-Design Bench or quietly keep reporting the old, flattering numbers.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Nous Releases Arete-27B, Cuts Qwen3.8-27B Propaganda From 22% to 0.9%
From Archives AI auditors detect 80% of agent misbehavior but trigger false alarms
Common Questions Answered
What is the performance gap between GPT-6 Astra's requirement completion rate and its fully correct task completion rate?
GPT-6 Astra, the best model tested, meets approximately 70 percent of a task's stated requirements on average, but achieves fully correct results in only 26.4 percent of tasks. This significant gap reveals that while the model handles the bulk of the work, it typically leaves at least one critical detail unfinished, requiring human intervention to complete the job.
Why did researchers from Alibaba create the Co-Design Bench benchmark?
Researchers developed Co-Design Bench specifically because older benchmarks had become ineffective at measuring real-world coding agent performance. The older tests allowed agents that could run their own code, check it, and patch mistakes to achieve artificially high scores, masking their actual limitations in practical applications.
How does the Co-Design Bench benchmark differ from older coding agent tests?
Co-Design Bench measures whether coding agents can satisfy actual user requirements with real-world scenarios, such as changing an existing web app based on specific instructions or screenshots. In contrast, older benchmarks essentially allowed models to study with the answer key available, measuring performance against a grader rather than against practical user needs.
What does the discrepancy between old benchmark scores and Co-Design Bench results reveal about AI coding agents?
The gap between 99 percent scores on older benchmarks and 25 percent completion rates on Co-Design Bench demonstrates that coding agents have been optimized for test-taking rather than real-world problem-solving. This reveals that current AI coding agents cannot reliably complete complex coding tasks without significant human cleanup and oversight.
