Editorial illustration for SupportNet Uses Largest k Scores to Route Queries for Faster, Efficient RAG
SupportNet Speeds RAG with Top-K Vector Routing
SupportNet Uses Largest k Scores to Route Queries for Faster, Efficient RAG
Vector search engines power most retrieval-augmented generation systems by running maximum inner product search, or MIPS, across databases that can hold millions of candidate documents. The setup is simple on paper: take a query vector, compare it against every key vector in the database, and return the closest match to feed into the language model. In practice, that comparison gets expensive fast. Once vector dimensions climb into the thousands and the candidate pool reaches millions of entries, exhaustive search turns into a bottleneck no production system can afford.
Engineers have leaned on clustering and quantization to build faster, approximate versions of this search. Those methods work by studying the structure of the keys themselves, the documents sitting in the database, and building index structures around that geometry. The problem is what they leave out.
A RAG pipeline doesn't just have keys, it has queries, and those queries come from actual users typing short, often messy questions. A method tuned only to the shape of long, curated reference documents has no particular reason to perform well when the input looks nothing like that.
Amortized MIPS reflects a shift in how retrieval can be approached when query distributions are known in advance â a common condition in production ML systems. By encoding MIPS solutions directly into network weights, the approach trades a one-time training (so-called amortization) cost for fast, lightweight inference and opens a practical path toward learned retrieval systems explicitly tailored to the queries they serve.
Why this matters
Anyone running RAG at scale knows the retrieval step is where latency and cost pile up, and most clustering-based speedups rely on crude centroid comparisons that miss the best match hiding in a less obvious cluster. SupportNet's pitch, training a model to predict the largest inner-product score a cluster could produce for a given query, is a more honest proxy than "how close is this query to the average point." For teams building search or recommendation pipelines on top of MIPS, that distinction matters directly: it's the difference between routing a query to a cluster that merely looks relevant and one that actually contains the best answer. We'd want to see how this holds up against standard approximate nearest neighbor baselines like IVF or HNSW on real workloads, not just synthetic benchmarks, before treating it as a drop-in replacement.
But the framing, asking a model to answer "how good is the best key here" rather than approximating distance, is the kind of small architectural shift that tends to matter more than the headlines around it suggest. Worth tracking if benchmarks against production-grade indexes follow.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google to End Free Access to Gemini AI for Non-Subscribers
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Enterprises Pair Smaller Language Models With LLMs in Tiered Architectures
From Archives Anthropic adds Claude bank access via Plaid to track spending
Common Questions Answered
How does SupportNet improve upon traditional clustering-based speedups for maximum inner product search?
SupportNet trains a model to predict the largest inner-product score a cluster could produce for a given query, which is a more accurate proxy than traditional centroid comparisons. Unlike crude centroid-based approaches that often miss the best match hidden in less obvious clusters, SupportNet's method provides a more honest representation of cluster potential, leading to better retrieval accuracy at scale.
What is amortized MIPS and how does it benefit production ML systems?
Amortized MIPS encodes maximum inner product search solutions directly into network weights, trading a one-time training cost for fast, lightweight inference during deployment. This approach is particularly effective in production ML systems where query distributions are known in advance, enabling learned retrieval systems that are explicitly tailored to the queries they serve.
Why does the retrieval step become expensive in vector search engines with large databases?
The retrieval step becomes expensive because it requires comparing a query vector against every key vector in a database that can hold millions of candidate documents. When vector dimensions climb into the thousands and the candidate pool reaches millions of entries, the computational cost of running maximum inner product search across all comparisons grows significantly.
What problem does SupportNet solve for teams running retrieval-augmented generation at scale?
SupportNet addresses the latency and cost issues that accumulate during the retrieval step of RAG systems at scale. By using largest k scores to route queries more efficiently through clusters, it reduces computational overhead compared to traditional clustering approaches, making it particularly valuable for teams building search and recommendation pipelines.
Further Reading
- Faster Retrieval for Better, More Efficient RAG - Apple Machine Learning Research
- Amortizing Maximum Inner Product Search with Learned Models - Apple Machine Learning Research
- Apple speeds up RAG with optimized vector search - Le Petit Récap
- RAG Architecture: Chunking, Embedding & Hybrid Retrieval - My AI Hardware
- Faster Retrieval for Better, More Efficient RAG - daily.dev
