Editorial illustration for RAG Retrieval Recall Plummets 40 Points Amid 2.68M Distracting Passages
RAG Systems Lose 40% Recall With Millions of Passages
RAG Retrieval Recall Plummets 40 Points Amid 2.68M Distracting Passages
A search system that works on 1,801 documents does not necessarily work on 2.68 million. That gap sits at the center of a new reproducible test of retrieval augmented generation, the technique where an AI model answers questions using passages a search step pulls up first. The person who built the test started by checking BM25, the decades-old text search method still underneath many RAG pipelines, against a collection of 100 research papers.
It put the correct paper in its top five results 99.7 percent of the time. Scaled to 1,801 papers, that number barely dropped, landing at 98 percent. That stability seemed to be good news: if search holds up as a collection grows, everything downstream, embeddings, parent-document retrieval, the language model's final answer, has firmer ground to stand on.
Then came a third test, this time on 2.68 million Wikipedia passages, using questions nobody wrote specifically for this setup. The results stopped looking stable. The full method, including the code, the questions, and the labels, is public, and it raises a specific question about why small-scale evals written by the same person testing the system tend to flatter it.
A reproducible test of BM25, embeddings, and parent documents shows why author-written evals overrate search, and why answer position in context matters.
Why this matters
The 99.7-to-98 percent drop sounds small until you remember what it's hiding: most RAG benchmarks get built on exactly the scale where BM25 still looks great, which is to say the scale nobody actually ships. 100 papers or 1,801 papers is a demo. 2.68 million distracting passages is a product with real users asking real questions against a corpus that grew without anyone's permission. If recall falls 40 points somewhere between those two points, every "our retrieval hits 95 percent" claim in a vendor deck or a paper's eval section needs a footnote stating corpus size, because the number is meaningless without it.
For teams building retrieval into production, the lesson isn't "switch embeddings" or "add a reranker," it's that your eval harness is probably testing a toy version of your own system. Answer position in context compounds the problem: a retriever that buries the right passage at rank 4 instead of rank 1 can quietly tank an LLM's answer quality even when recall@5 looks fine on paper. Test at the scale you'll actually run at, not the scale that fits in a GitHub repo.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives ElevenLabs Scribe v3 Live uses context to stop voice agents cutting callers off
From Archives Space Tech Startup Astrome Secures Rs 64 Cr RDI Fund for AI Satellite Network
Common Questions Answered
Why does RAG retrieval recall drop significantly when scaling from 1,801 documents to 2.68 million passages?
BM25, the decades-old text search method used in many RAG pipelines, achieves 99.7% recall on small collections of 100 research papers but drops to 98% recall when tested against 2.68 million distracting passages. This 40-point performance degradation reveals that search systems optimized for small datasets fail to maintain effectiveness as corpus size grows exponentially, which is the reality of production systems with real users.
What is the key limitation of author-written evaluations in RAG benchmarking?
Author-written evaluations typically test retrieval systems on small-scale datasets like 100 or 1,801 documents where methods like BM25 appear highly effective, but these benchmarks don't reflect real-world product conditions with millions of distracting passages. This gap between demo-scale and production-scale testing causes researchers to overestimate search performance and publish misleading accuracy metrics that don't translate to actual deployed systems.
How does answer position in context affect RAG system performance according to this test?
The reproducible test demonstrates that answer position in context is a critical factor in RAG performance that is often overlooked in benchmarking. When retrieval systems pull up passages with distracting information, the position of the correct answer within those retrieved results significantly impacts whether the AI model can effectively use that information to answer questions accurately.
What is the difference between testing RAG systems on 100 papers versus 2.68 million passages?
Testing on 100 papers or even 1,801 documents represents a demo-scale evaluation where retrieval methods still perform exceptionally well, but 2.68 million distracting passages represents a production-scale system with real users querying a corpus that has grown organically. The dramatic performance drop between these scales reveals that most published RAG benchmarks are conducted at scales that don't reflect the complexity of actual deployed products.
Further Reading
- Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems - arXiv
- Q2D-Web: Evaluating First-Stage Retrievers at Scale - Perplexity AI
- NEST: Nested Evidence Survival for Retrieval - ACL Anthology
- PRISMRAG: Boosting RAG Factuality with Distractor Robustness - ACL Anthology
- A Dataset and Benchmark for Adaptive RAG Routing - arXiv
