Skip to main content
Gemini API's new tool retrieval and prefix caching features are represented by a stylized, glowing brain icon connected to va

Editorial illustration for Gemini API Adds Tool Retrieval to Work With Prefix Caching

Gemini API Adds Tool Retrieval for Scaled Agents

Gemini API Adds Tool Retrieval to Work With Prefix Caching

• 4 min read

Google added Tool Retrieval to the Gemini API on preview in the Interactions API, aimed at a problem that shows up once agents scale past a handful of functions. Connect an agent to dozens or hundreds of tools, and sending every parameter schema with every request starts costing real money and real time: token counts balloon, latency climbs, and the model gets worse at picking the right tool as the candidate list grows.

The fix Google landed on keeps the full tool library visible to the model in name and description only, while holding the detailed parameter schemas back until they're actually needed. Developers turn this on by adding a tool_search entry to their request and flagging specific tools with defer_loading: true. The model can see that a tool like issue_refund exists and what it roughly does, but it doesn't load the full schema, order_id, amount, required fields, and so on, unless the task calls for it.

That setup raises an obvious question: how does the model actually fetch a deferred tool's schema mid-request, and what does that lookup-and-call sequence look like in practice for something like a damaged-order refund.

When an agent connects to dozens or even hundreds of tools, sending every full parameter schema on every request adds up fast. It inflates input token counts, adds latency, and lowers tool-selection accuracy as the candidate list grows. Tool Retrieval lets your agent keep its full tool library on standby and load parameter schemas only when the current task needs them.

Why this matters

For anyone building agents against the Gemini API, this is a practical fix for a problem that gets worse as tool libraries grow. Dumping every parameter schema into the prompt on every call was never going to scale past a handful of tools, and it was quietly taxing both latency and tool-selection accuracy as the candidate list expanded. By loading schemas only when a tool is actually needed, and keeping those deferred schemas out of the prompt's start, Google is letting developers preserve prefix caching rather than breaking it with bloated, shifting inputs.

That matters for anyone running cost-sensitive production agents, since cache hits on the prefix translate directly into lower token spend and faster responses. The added support for remote MCP servers suggests Google wants this to work for teams assembling tools from outside sources, not just first-party functions. Preview status means we should watch for rough edges, but the design choice (retrieval that respects caching mechanics instead of ignoring them) is the kind of plumbing work that determines whether "agent with hundreds of tools" is a demo or something you can actually run in production.

Common Questions Answered

What problem does Tool Retrieval solve for agents using the Gemini API?

Tool Retrieval addresses the scaling challenge that occurs when agents connect to dozens or hundreds of tools, where sending every parameter schema with each request causes token counts to balloon, increases latency, and reduces the model's accuracy in selecting the correct tool. By loading parameter schemas only when needed rather than including all of them in every request, Tool Retrieval keeps the full tool library accessible while maintaining performance efficiency.

How does Tool Retrieval reduce input token counts in the Gemini API?

Tool Retrieval keeps the full tool library on standby and loads parameter schemas only when the current task requires them, rather than sending every full parameter schema on every request. This approach prevents the inflation of input tokens that occurs when all tool information is included in the prompt from the start, making the system more cost-effective and faster.

Why does tool-selection accuracy decrease as the candidate list grows without Tool Retrieval?

When agents have access to dozens or hundreds of tools and all parameter schemas are sent with every request, the expanded candidate list makes it harder for the model to accurately identify and select the most appropriate tool for a given task. Tool Retrieval improves this accuracy by reducing cognitive load on the model through deferred schema loading, allowing it to focus more effectively on selecting the right tool.

What is the relationship between Tool Retrieval and Prefix Caching in the Gemini API?

Tool Retrieval was added to the Gemini API on preview in the Interactions API and works in conjunction with Prefix Caching to optimize performance. By deferring parameter schemas and keeping them out of the prompt's start, Tool Retrieval complements caching mechanisms to improve both latency and efficiency for agents managing large tool libraries.

LIVE20:50AI token spending growth slows: Anthropic 0.5%, OpenAI 2.1%, US buyers 56.8%