Editorial illustration for RadixAttention Speeds First Tokens by Reusing Cached KV States
RadixAttention Speeds First Tokens With KV Cache
RadixAttention Speeds First Tokens by Reusing Cached KV States
A Llama-3 8B model holding a 100,000-token conversation in memory needs close to 12.8 GiB just for its KV cache, before a single additional request gets queued. That number, not the model's weights, is often what decides how many users a GPU can serve at once. Every transformer generates tokens one at a time, and each new token has to attend back to every key and value vector that came before it.
Recomputing those vectors from scratch at each step would be too slow for real use, so serving engines cache them. That cache saves computation, but it scales linearly with sequence length, and on long-context workloads it becomes the biggest dynamic consumer of GPU memory in the whole system.
Two engineering approaches have taken on this problem from different angles. PagedAttention borrows a trick from operating systems, breaking the cache into fixed-size blocks so memory gets allocated without waste or fragmentation. RadixAttention takes aim at redundancy instead, letting requests that share a common prefix, like a system prompt, reuse the same cached keys and values rather than recomputing them. Both change how much a GPU can actually serve.
Modern LLMs rely on quantization, pruning, distillation, and faster attention kernels, but production performance often depends most on KV cache management. As context windows grow, the cache consumes significant GPU memory, limiting concurrency, throughput, and latency. Two breakthroughs transformed this challenge: PagedAttention improves memory allocation, while RadixAttention enables efficient prefix reuse.
Why this matters
For anyone running LLMs in production, this is a reminder that the flashy optimizations, quantization, pruning, distillation, aren't where the real serving costs hide. They hide in the KV cache. RadixAttention's trick of skipping redundant transformer computation on shared prefixes is a narrow fix, but it targets the exact metric users notice first: how long they wait before anything appears on screen.
That's a different problem than PagedAttention solves, and conflating the two leads teams to optimize the wrong thing. If you're a founder pricing API calls or a developer tuning a serving stack, the practical question is which of your workloads actually share prefixes, chat histories, system prompts, few-shot templates, because that's where RadixAttention pays off and where it does nothing. Researchers building the next serving framework should take note that TTFT and memory efficiency are separate axes requiring separate engineering, not a single "faster attention" checkbox.
The pairing of these two techniques suggests serving infrastructure is maturing past one-size-fits-all kernels into more deliberate, workload-aware caching strategies, worth watching as context windows keep expanding.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Deepseek Targets Visual Agents With New Experimental Flash Model
From Archives Meta Pays Microsoft Hundreds of Millions Annually for AI
Common Questions Answered
Why does KV cache management matter more than model quantization or pruning in production LLM serving?
KV cache consumption is the primary bottleneck that determines how many concurrent users a GPU can serve, often consuming more resources than the model weights themselves. While optimizations like quantization and pruning improve model efficiency, the real serving costs are hidden in KV cache memory usage, which directly limits concurrency, throughput, and latency at scale.
How much GPU memory does a Llama-3 8B model require just for its KV cache in a 100,000-token conversation?
A Llama-3 8B model holding a 100,000-token conversation in memory requires close to 12.8 GiB just for its KV cache, before any additional requests are queued. This substantial memory requirement demonstrates why KV cache management is critical for production deployment and user concurrency.
What is the key difference between how RadixAttention and PagedAttention address KV cache challenges?
PagedAttention improves memory allocation efficiency for KV caches, while RadixAttention enables efficient prefix reuse by skipping redundant transformer computation on shared prefixes. RadixAttention specifically targets the first-token latency that users notice immediately, whereas PagedAttention addresses overall memory management and concurrency.
Why do transformers need to cache key and value vectors instead of recomputing them at each step?
Each new token generated by a transformer must attend back to every key and value vector that came before it, and recomputing those vectors from scratch at each step would be too slow for real-time production use. Caching these vectors allows for faster inference, though it comes at the cost of significant GPU memory consumption.
How does RadixAttention improve the user experience in LLM serving?
RadixAttention reduces first-token latency by reusing cached KV states from shared prefixes, which directly impacts the metric users notice first: how long they wait before anything appears on screen. This optimization targets a different problem than memory allocation improvements, focusing specifically on the perceived responsiveness of LLM applications.
Further Reading
- SGLang: Efficient Execution of Structured Language Model Programs - arXiv
- Prompt Cache: Modular Attention Reuse for Low-Latency Inference - arXiv
- SCBench: A KV Cache-Centric Analysis of Long-Context LLM Serving - ICLR 2025 Poster
- KV cache memory calculator: how much does your LLM actually use? - DEV Community
- HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading - arXiv
