Editorial illustration for Google's ToolGrad Framework Hits 99.8% Pass Rate for Tool-Use Data
Google's ToolGrad Hits 99.8% Pass Rate for API Calls
Building datasets that teach language models to call APIs correctly has always run into the same wall: you ask a model to dream up a plausible user request, then send a search agent hunting for a tool chain that satisfies it. Most of those searches fail, and the compute behind them gets thrown away. Google Research, working with the University of Tokyo, RIKEN AIP, and Tohoku University, just published a framework called ToolGrad that flips the sequence entirely.
Instead of starting with a guessed query and searching for a matching chain, ToolGrad builds a verified, working tool-use chain first by actually running the APIs, then generates the user query to fit it afterward. That ordering turns an unreliable search problem into a single, well-defined annotation step. The team fine-tuned Gemma-3 models on just 500 samples of the resulting data and posted results on the Berkeley Function Calling Leaderboard that land near frontier proprietary systems.
Code, dataset, and model weights are already public. Here's what the researchers say breaks down about the older approach.
ToolGrad reverses the order. It first constructs a ground-truth tool-use chain by actually executing APIs, then annotates that chain with a matching user query. An explicit, working chain is far less ambiguous than a hypothetical prompt, so the chain-to-query step takes a single LLM call.
Why this matters
For anyone building agents that call APIs, the bottleneck has never been model architecture, it's been getting clean training data without paying an army of annotators or burning through DFS-based sampling that produces broken chains. ToolGrad's answer-first inversion is a simple idea that apparently works: verify the tool chain before you even write the query, and you skip most of the failure modes that plague ToolBench-style generation. A 99.8% pass rate against a 16,000-API database is a real number worth checking against your own pipelines, not a marketing figure.
The Gemma-3 results on just 500 samples matching frontier proprietary models should catch the attention of teams who assumed they needed massive datasets to compete. We'd still want to see this tested outside ToolBench's own API set before treating it as settled. But the default config, 10 iterations over 50 sampled APIs per workflow, is a good starting point for anyone running fine-tuning experiments this quarter.
The open question is whether this approach holds up when the tool universe gets messier than a curated benchmark.
Related Reading
Browse all Google news →
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
Related Topic NVIDIA and Google Cloud let developers scale AI from prototype to production
Market Trends
Related Topic Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI
From Archives OpenDiscoveryTrace Records 124 AI Scientist Workflows With 9-Step Detail
From Archives Robotaxi Leaders Build With NVIDIA’s Physical AI Tools
From Archives Deepmind Eased Ban on Staff Discussing AI Risks After Internal Pushback
From Archives Google Open-Sources Mantis Toolkit for Automated Vulnerability Patches
Open Source
Common Questions Answered
How does ToolGrad's answer-first approach differ from traditional tool-use dataset generation?
ToolGrad reverses the conventional sequence by first constructing a ground-truth tool-use chain through actual API execution, then annotating that chain with a matching user query. This is fundamentally different from the traditional approach that starts with a guessed user request and searches for a tool chain to satisfy it, which often results in failed searches and wasted compute resources.
What pass rate did ToolGrad achieve in its testing, and how does it compare to previous methods?
ToolGrad achieved a 99.8% pass rate when tested against a 16,000-API benchmark. This represents a significant improvement over traditional ToolBench-style generation methods, which suffer from numerous failure modes that ToolGrad's answer-first inversion successfully avoids.
Why is ToolGrad's explicit, working tool chain more effective than hypothetical prompts for LLM training?
An explicit, working tool chain is far less ambiguous than a hypothetical prompt because it represents an actual, verified execution path through APIs. This clarity allows the chain-to-query annotation step to be completed with just a single LLM call, making the dataset generation process more efficient and reliable.
What organizations collaborated on developing the ToolGrad framework?
Google Research developed ToolGrad in collaboration with the University of Tokyo, RIKEN AIP, and Tohoku University. This multi-institutional partnership combined expertise to create the framework that addresses the critical bottleneck in building agents that call APIs.
What was the primary bottleneck in building API-calling agents before ToolGrad?
The main bottleneck was obtaining clean training data without either paying large teams of human annotators or using DFS-based sampling that frequently produced broken tool chains. ToolGrad solves this problem by verifying tool chains before writing queries, eliminating most failure modes that plagued previous generation methods.
Further Reading
- ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients" - ACL Anthology
- ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients" - arXiv
- ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients" - Project page
- zhongyi-zhou/toolgrad-12b - Hugging Face
- Benchmarking Tool Retrieval for Large Language Models - arXiv
