Editorial illustration for Keyword Search Tied an 8B Embedding Model at Picking Agent Tools
8B Embedding Model Matches Keyword Search at Tool Retrieval
Keyword Search Tied an 8B Embedding Model at Picking Agent Tools
A developer ran a test that most teams building AI agents skip: what happens to tool retrieval as the tool count actually scales. The setup used 200 questions from ToolRet, a public benchmark for tool retrieval, and held the correct answers fixed while the surrounding pile of tools grew from 1,000 to 3,794. Two approaches went head to head.
One was BM25, a decades-old keyword search algorithm running on plain CPU. The other was BGE-large, an embedding model built for exactly this kind of semantic matching. The question driving the comparison was simple and rarely asked out loud: once an agent has too many tools to stuff into a prompt and has to switch to retrieval, does that retrieval actually hold up as the number of tools climbs, and does spending more on a bigger model buy you anything.
MCP servers have made it trivial to hand an agent hundreds of tools at once, which means picking the right one on every request is no longer a trivial problem. What this test measured was whether the standard advice, retrieve once you can't fit everything in context, actually survives contact with a few thousand tools.
Every method got worse as the pile grew, and by roughly the same amount, about a fifth of its score. A plain keyword search from the 1990s kept up with an embedding model 24 times its size while running hundreds of times faster. Putting every tool in the prompt ran out of room long before the pile stopped growing.
Why this matters
If you're building agents that call tools, this result should change your default. The instinct to reach for a fine-tuned retriever or a bigger embedding model is understandable, but the data here says BM25 on a CPU matched an 8B model while running 40x faster, and it did so without any retraining when you add a new MCP server. That's a real operational cost difference, not a marginal one.
Every team currently maintaining a fine-tuned retrieval layer for tool selection should ask whether that complexity is buying anything, especially once tool counts climb into the hundreds. There's a broader lesson too: more tools crammed into an agent's drawer doesn't just slow retrieval, it degrades accuracy across the board, which argues for curating tool sets rather than assuming retrieval will sort it out. For founders shipping agent products, this is a cost and latency lever you can pull today.
For researchers, it's a reminder that boring baselines deserve a seat at the table before anyone claims a fancier model is pulling its weight.
Related Reading
Open Source Study: 6.7% of Deepfake Requests on Hugging Face Targeted Children
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
Open Source Axiom Math raises USD 200M Series A led by Menlo to keep AI slop out of math
From Archives Perplexity Releases Decider v1.3 With Locate, Self-Reported 91.6% Accuracy
From Archives Liquid AI Releases LiquidCodec, a 74M-Parameter Streaming Audio Codec
Common Questions Answered
How did BM25 keyword search perform compared to the BGE-large embedding model when tool count scaled to 3,794?
BM25 keyword search kept pace with the 8B BGE-large embedding model while running 24 times faster and hundreds of times faster on CPU. Both methods experienced approximately a fifth decline in their scores as the tool pile grew, demonstrating that the decades-old algorithm matched the modern embedding model's performance at scale without requiring retraining.
What was the ToolRet benchmark used to measure in this tool retrieval test?
ToolRet is a public benchmark for tool retrieval that provided 200 questions used to test how different retrieval methods perform as tool count scales. The test held the correct answers fixed while progressively increasing the surrounding pool of tools from 1,000 to 3,794 to measure real-world performance degradation.
Why does the article suggest teams should reconsider their default approach to tool selection for AI agents?
The test results show that BM25 keyword search matched an 8B embedding model's performance while running 40 times faster and without requiring any retraining when new tools are added. This represents a significant operational cost difference, making BM25 a more practical default choice than maintaining fine-tuned retrieval layers or larger embedding models for tool selection.
What limitation did putting every tool directly in the prompt face according to the test results?
Putting every tool in the prompt ran out of room long before the tool pile stopped growing at 3,794 tools. This approach became impractical as the number of available tools increased, highlighting the need for effective tool retrieval methods rather than exhaustive enumeration.
Further Reading
- Benchmarking Tool Retrieval for Large Language Models - arXiv
- Bridging Tools and Agents for Scalable LLM Multi-Agent Systems - arXiv
- Scaling Enterprise Agent Routing: Degradation, Diagnosis, and ... - arXiv
- LLM-Native Recursive Construction and Search of Service ... - arXiv
- ScaleCall: Agentic Tool Calling at Scale for Fintech: Challenges, Methods, and Deployment Insights - arXiv
