Editorial illustration for IndexCache sparse attention optimizer makes long-context AI 1.82× faster
IndexCache Slashes Long-Context AI Inference Time 1.82×
IndexCache sparse attention optimizer makes long-context AI 1.82× faster
Long-context AI models have a problem with the paperwork. The real computational work of figuring out which words matter to which other words has already been solved, more or less. Dynamic sparse attention made that core task efficient.
But before the model can even start that work, a smaller, dumber piece of code called the indexer has to run. It labels the data. It's a lighter job, but it happens at every single layer of the model, and its cost scales poorly.
As you feed the model more text—a book instead of a paragraph—this administrative chore becomes the dominant expense, especially in the initial "prefill" phase. IndexCache fixes this by making the indexer do its job once, then remember it. The result is a 1.82x speedup on long-context inference.
Not a small tweak. A structural fix.
Researchers at Tsinghua University and Z.ai have built a technique called IndexCache that cuts up to 75% of the redundant computation in sparse attention models, delivering up to 1.82x faster time-to-first-token and 1.48x faster generation throughput at that context length.
The improvement is practical, not just theoretical. It means the next wave of long-context models won't just be capable of processing a novel's worth of text. They'll be able to do it at a cost that doesn't make the exercise pointless.
This cuts the overhead of ambitious applications in half. Analyzing a legal corpus, tracing a multi-threaded conversation, reasoning across an entire software repository—these tasks just became significantly cheaper to run. The bottleneck wasn't the thinking.
It was the bureaucracy required to enable the thinking. IndexCache removes it.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Apple's Siri to support third‑party AI chatbots via new 'Extensions' feature
From Archives Import AI memories and chat history to Gemini for seamless continuity
Common Questions Answered
How does IndexCache improve the performance of long-context language models?
IndexCache optimizes sparse attention by reducing the computational complexity of the DSA indexer from quadratic to linear. This approach cuts up to three-quarters of wasted compute, resulting in a 1.82× boost to time-to-first-token and a 1.48× improvement in generation throughput when processing 200,000 tokens.
What was the key limitation in the original Dense-Sparse Attention (DSA) architecture?
While DSA successfully reduced core attention computation from quadratic to linear complexity, the DSA indexer itself still operated at a quadratic complexity at every layer. As context lengths increased, the time spent running these indexers would dramatically increase, creating a performance bottleneck.
What performance gains does IndexCache achieve when integrated with DeepSeek Sparse Attention?
IndexCache delivers a 1.82× improvement in time-to-first-token and a 1.48× lift in generation throughput when processing 200,000 tokens. Importantly, the optimizer maintains the linear-scaling core attention of DSA while preserving the overall output quality of the model.
Further Reading
- Accelerating Sparse Attention via Cross-Layer Index Reuse — arXiv
- IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse — Hugging Face Papers
- IndexCache: Faster Sparse Attention for LLMs — YouTube - AI Research Roundup
- IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse — GitHub - THUDM
