Editorial illustration for Keenable AI Open-Sources NEEDLE, a Live Search Benchmark With Hourly Updates
NEEDLE: Live Search Benchmark Stops AI Cheating
Keenable AI Open-Sources NEEDLE, a Live Search Benchmark With Hourly Updates
Search benchmarks have a data leakage problem that nobody built around until now. Give an agent a fetch tool and a public dataset with fixed gold labels, and it can just download the answer key mid-evaluation, no retrieval required. Even without cheating, static benchmarks fail differently: if a model already has the answer baked into its training data, getting the question right proves nothing about whether its search tool actually works.
Keenable AI's response is NEEDLE, a benchmark that refuses to sit still. Instead of freezing a query set once and reusing it, NEEDLE regenerates its questions on a schedule tied to how fast each domain actually changes. News queries pull fresh from RSS feeds and Google Trends every hour.
Finance, legal, scholarly, and rare-entity queries rebuild daily from sources like SEC filings, arXiv, CourtListener, and Europe PMC. Fifteen search APIs get tested against identical query text, scored against a pooled oracle called "ultimate" that tracks what the best available search across all engines could find. The whole thing ships as an open-source Python CLI, not a hosted product, meant to run on a laptop or inside CI.
For each query, NEEDLE pools the results returned by every engine into a synthetic oracle engine, then orders that combined set by relevance. That creates an empirical ceiling based on what the entire field was able to retrieve.
Why this matters
Anyone building or buying a search agent right now is stuck grading homework the model itself might have peeked at. NEEDLE's hourly refresh cycle is a direct fix for that: if the query set didn't exist an hour ago, there's no answer key sitting in a crawl for the agent to fetch or a fact baked into pretraining data. That's a meaningfully different bar than the static benchmarks most teams still cite in launch posts.
The scoring split matters too. nDCG@5 with a duplicate-URL penalty for news, answer-recall@5 for finance, identifier matching for scholar and legal, each domain gets graded on what it's actually supposed to prove, not a single blended score that hides where an agent is faking retrieval. For founders shipping search products, this is worth adopting before a customer asks why your "live web" claim scored well on a benchmark built last year.
For researchers, it's a cleaner testbed for measuring whether retrieval is doing real work or just window dressing on a model that already knew the answer.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Insurance Company's 'AI Jim' Chatbot Handled Initial Claims Reports
From Archives AI-Powered Robot Learns How to Assist Stroke Patients in Therapy
Common Questions Answered
What data leakage problem does NEEDLE solve in search benchmarks?
Traditional search benchmarks suffer from data leakage because agents can download answer keys from public datasets during evaluation, and models may already have answers baked into their training data. NEEDLE addresses this by using hourly-updated query sets that didn't exist before, preventing agents from accessing pre-existing answer keys or relying on pretraining data to answer questions correctly.
How does NEEDLE's synthetic oracle engine work to establish relevance scoring?
For each query, NEEDLE combines the results returned by every search engine into a single pool and orders them by relevance to create an empirical ceiling. This approach shows what the entire field of search engines was collectively able to retrieve, providing a fair and comprehensive benchmark for evaluating search agent performance.
Why is NEEDLE's hourly refresh cycle important for evaluating search agents?
The hourly refresh cycle ensures that query sets are constantly updated and didn't exist in previous data crawls or training datasets. This prevents search agents from cheating by fetching answer keys or relying on memorized information, creating a meaningfully different and more rigorous evaluation bar than static benchmarks that most teams currently use.
What advantage does NEEDLE provide over static benchmarks for search agent evaluation?
Unlike static benchmarks where models might have already learned the answers during training or agents could download answer keys, NEEDLE's live updating mechanism ensures fair evaluation of actual search tool functionality. The benchmark prevents both data leakage and training data contamination, making it a more reliable measure of whether a search agent's retrieval capabilities actually work.
Further Reading
- NEEDLE: The benchmark your search engine can't memorize - Keenable.ai
- How does Keenable's NEEDLE benchmarking suite evaluate web search APIs for agent-like queries - The Agentic Digest
- Search-Time Data Contamination - arXiv
- Search-Time Contamination in Deep Research Agents - arXiv
- Accel-backed Keenable is indexing the web for AI agents - Keenable.ai
