Editorial illustration for Two-Stage RAG Pipeline Uses Initial LLM Call to Match TOC Sections
Two-Stage RAG Pipeline Uses Initial LLM Call to Match...
Most retrieval-augmented generation is just expensive keyword search. It fails when the document gets too big.
The problem is noise. Asking a model to find a specific clause in a 15,000-line contract is hopeless. The answer gets buried.
A new pipeline suggests a simple fix: use one LLM call to decide where to look, and another to actually look there. First, the model scans a table of contents and picks the two or three relevant sections. Then, a keyword search runs only on those few hundred lines.
It costs more upfront. But when you're dealing with a 500-page filing, that initial filter is the only thing that works.
4.1 Reason-then-match (two-LLM-call alternative) A two-stage pipeline that uses an extra LLM call up front: the LLM reads the TOC, picks the relevant sections, returns a short list of section IDs, then keyword retrieval runs only on the lines within those sections. When this is worth the extra call: A 100-page contract has 50 sections; the LLM picks 2 to 3 in one call; keyword retrieval then operates on a few hundred lines instead of the full 15,000. The trade-off versus the single-arbiter pattern: you pay two LLM calls instead of one, but the second-stage keyword search runs over a much smaller pool, which matters when the pool is huge (think: a 500-page regulatory filing).
This isn't a minor tweak. It changes what the system is doing. Instead of one model drowning in context, you have a scout and a specialist.
The first call isn't wasted money. It's what makes the second call useful at all. The efficiency comes from admitting most of any long document is irrelevant.
Find the right chapter first. Then search the page. It works because it's obvious.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Harness-1 20B Model Beats GPT-5.4, Curates Top 8 Fairness‑Rated Results
From Archives DFlash drafts whole token blocks, achieving 15× throughput on NVIDIA Blackwell
Common Questions Answered
How does the two-stage RAG pipeline improve upon traditional retrieval-augmented generation systems?
The two-stage RAG pipeline uses an initial LLM call to scan a table of contents and identify the two or three most relevant sections before conducting a keyword search. This approach eliminates the noise problem that occurs when asking a model to search through massive documents like 15,000-line contracts, where the answer gets buried in irrelevant context. By separating the task into a scout phase and a specialist phase, the system dramatically improves efficiency and accuracy.
Why does traditional keyword search fail when documents become too large?
Traditional keyword search fails on large documents because it generates excessive noise, making it nearly impossible for a model to locate specific clauses or information within thousands of lines of text. The answer becomes buried under irrelevant results, making the retrieval process hopeless. The two-stage pipeline solves this by first narrowing down the search space to relevant sections before applying keyword search.
What is the role of the first LLM call in the two-stage RAG pipeline?
The first LLM call acts as a scout that scans the document's table of contents and identifies the two or three most relevant sections for the query. This initial call is not wasted effort but rather makes the second call useful by drastically reducing the context that needs to be searched. It changes the system from one model drowning in context to a specialized approach that admits most of any long document is irrelevant.
How does the two-stage RAG approach change the efficiency model compared to single-stage retrieval?
Instead of one expensive LLM call attempting to search through an entire massive document, the two-stage approach uses two focused calls where the first identifies relevant chapters and the second searches only those specific pages. This separation of concerns means the efficiency gains come from admitting that most of a long document is irrelevant and finding the right chapter first before searching the page. The system transforms from a brute-force approach into a strategic two-phase process that reduces noise and improves accuracy.
Further Reading
- Anchor Detection for RAG: Parallel Detectors, Then One LLM Call at the End — Towards Data Science
- Zero-Shot Document Understanding using Pseudo Table of Contents — arXiv
- Boost RAG Accuracy with Two-Stage Retrieval Pattern — LinkedIn
- Two-Stage Multi-Pass Retrieval for Context Retention in Azure AI Search RAG Pipelines — gopenai Blog
- Common Challenges in RAG and How to Solve Them in Production — Unstructured.io
