Skip to main content
Microsoft Open Source Test Agent outperforms Copilot, achieving 92.1% completion. AI, software development, tech news.

Editorial illustration for Microsoft's Open Source Test Agent Hits 92.1% Completion, Outperforming Copilot

Microsoft's Test Agent Hits 92.1%, Beats Copilot

4 min read

Microsoft has released code-testing-generator into open source, folding it into the MIT-licensed dotnet/skills repository as part of the dotnet-test plugin. The pitch is narrow but practical: coding assistants are good at writing code, bad at deciding how to test it. Ask Copilot to "generate unit tests" and it has to guess the framework, the file location, the assertion style, guesses that often don't match what the rest of the repo is doing. Microsoft's agent skips the guessing by reading the codebase first, then planning, writing, running and checking its own tests before handing anything back.

The numbers are the headline here. On Microsoft's internal benchmark of 152 tasks, the new agent finished 140 of them. Stock GitHub Copilot, running the same model and the same prompts, finished 120.

That gap, 92.1% completion against 78.9%, is what Microsoft is putting forward as the case for the tool. Because it's shipped as an agent definition rather than a hosted service, it runs inside whatever coding agent a team already uses, with code staying local rather than round-tripping through a third-party server.

On Microsoft’s internal benchmark of 152 tasks from real repositories, the agent completed 140 (92.1%) versus 120 (78.9%) for stock GitHub Copilot on the same model and prompts (63% fewer failures).

Why this matters

The 92.1% versus 78.9% gap matters less than how Microsoft closed it. Stock Copilot fails here because "write unit tests" is an underspecified prompt, and the model guesses at framework, file placement, and assertion style. code-testing-generator instead reads the repo first, then runs five verification checks including mutation-style reasoning about which code changes should break its own tests. That's a meaningful design shift: verification baked into the agent's workflow rather than left to a human reviewer after the fact.

For developers, the MIT license and the dotnet/skills packaging mean this is inspectable and forkable today, not a locked-away benchmark claim. Founders building dev tools should note where the real gains came from: not a bigger model, but a scaffolding of self-checks around a modest one. That's a cheaper, more portable lesson than "use a better LLM."

For researchers, the open question is whether this verification pattern generalizes past unit tests, to code review, refactoring, or documentation agents, or whether test generation just happens to have unusually clean success criteria that other coding tasks lack. Worth watching who forks this next and what they build on top of it.

Common Questions Answered

What is the key difference between Microsoft's code-testing-generator and stock GitHub Copilot?

Microsoft's code-testing-generator reads the repository first to understand existing testing patterns, frameworks, and assertion styles before generating tests, whereas stock Copilot must guess at these details. This approach eliminates the guessing that often results in tests that don't match the repository's conventions, leading to significantly fewer failures.

What completion rate did Microsoft's test agent achieve on its internal benchmark?

Microsoft's code-testing-generator achieved a 92.1% task completion rate (140 out of 152 tasks) on the internal benchmark from real repositories, compared to 78.9% for stock GitHub Copilot on the same model and prompts. This represents a 63% reduction in failures and demonstrates the effectiveness of the repository-aware approach.

How does code-testing-generator verify the quality of generated tests?

The agent runs five verification checks including mutation-style reasoning, which evaluates whether code changes should break the generated tests. This verification process is baked into the agent's workflow, ensuring that tests are not only syntactically correct but also logically sound and capable of catching actual bugs.

Where has Microsoft released the code-testing-generator and under what license?

Microsoft has released code-testing-generator as open source in the MIT-licensed dotnet/skills repository as part of the dotnet-test plugin. This makes the tool freely available for developers to use and contribute to.

What specific problems does code-testing-generator solve that traditional coding assistants struggle with?

Traditional coding assistants like Copilot struggle with deciding how to test code because they must guess at the testing framework, file location, and assertion style without repository context. Code-testing-generator solves this by analyzing the existing codebase first, eliminating guesswork and ensuring generated tests align with the project's testing conventions and standards.

LIVE07:55Microsoft's Open Source Test Agent Hits 92.1% Completion, Outperforming Copilot