Editorial illustration for TriAttention KV Cache Compression Matches Full Attention, 2.5× Faster
TriAttention: KV Cache Compression Boosts AI Speed
TriAttention KV Cache Compression Matches Full Attention, 2.5× Faster
The key-value cache is the bottleneck of modern long-context inference, growing without bound as sequences lengthen, slowing throughput to a crawl. Pruning it has always meant trading accuracy for speed. Until now.
TriAttention, a new compression method from researchers at MIT, NVIDIA, and Zhejiang University, shatters that tradeoff. It achieves 2.5× higher throughput while matching full-attention accuracy on exacting math benchmarks. The insight is elegant: across attention heads, 96.6% of queries and keys concentrate tightly around a shared center, a property that holds whether you use MLA or GQA.
TriAttention exploits this by scoring keys offline, before any query arrives, using a trigonometric series that predicts future attention based on positional distance. A norm-based score handles the remaining heads, and an adaptive weight blends the two. Every 128 tokens, the cache is trimmed to the top-B, no live query observations needed.
On MATH 500, with only 1,024 tokens cached out of 32,768, TriAttention scores 68.4% against full attention’s 69.6%. On AIME25 it beats the leading compression method by 15.4 percentage points. This isn’t a narrow trick, it’s a general principle, waiting to be deployed.
To understand why TriAttention is important, it helps to understand the standard approach to KV cache compression. Most existing methods — including SnapKV, H2O, and R-KV — work by estimating which tokens in the KV cache are important and evicting the rest. Importance is typically estimated by looking at attention scores: if a key receives high attention from recent queries, it is considered important and kept.
TriAttention doesn’t just compress the KV cache, it rethinks what compression should look like. By treating attention heads not as monolithic but as a spectrum of concentration behaviors, the method adapts: trigonometric scoring dominates where queries are predictable, norm-based scoring catches the outliers that matter. The results speak clearly.
On MATH 500, TriAttention retains 68.4% accuracy with a cache of just 1,024 tokens, against Full Attention’s 69.6% across 32,768. That’s a 32x compression with near-zero degradation. Against R-KV, the gap is brutal: 15.4 points on AIME25.
And the speedup? 2.5× throughput, no asterisk. This work also does something quieter but perhaps more important.
It confirms that Q/K concentration is not a quirk of one architecture, it’s a general property of modern LLMs. Once you see that, compression becomes a matter of knowing which keys will be seen, not guessing which queries will arrive. TriAttention nails that knowing.
The Recursive State Query benchmark adds a stress test that rewards methods with real retention over long dependencies. TriAttention passes. The field has been chasing lossless compression for years.
This is the first method that convincingly says: you don’t need full attention to get full accuracy. You just need to look at the data the right way.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Knowledge Distillation Keeps Student Model Capacity to Match Ensemble Boundaries
From Archives Google AI's PaperOrchestra boosts manuscript success, 79‑81% win rate
Common Questions Answered
How does TriAttention achieve KV cache compression without losing model performance?
TriAttention leverages the observation that modern large language models concentrate their query and key vectors in narrow subspaces. By using a trigonometric series scoring function that computes the key center offline, the method can prune and score keys without needing live query observations, effectively reducing cache size while maintaining full-attention quality.
What percentage of attention heads show high vector concentration across different attention designs?
In the Multihead Linear Attention (MLA) study, 96.6% of attention heads exhibit a high concentration ratio (R > 0.95), compared to 84.7% for Grouped Query Attention (GQA). This confirms that the concentration of query and key vectors is a general property across different attention mechanisms in modern large language models.
What performance improvements does TriAttention offer for KV cache compression?
TriAttention delivers a KV cache compression technique that boosts throughput by approximately 2.5 times while retaining the quality of full-attention models. By intelligently scoring and pruning keys based on their vector concentration, the method can significantly reduce cache size without compromising the model's accuracy or long-context performance.
Further Reading
- Efficient Long Reasoning with Trigonometric KV Compression — arXiv
- Efficient Long Reasoning with Trigonometric KV Compression — arXiv
- TriAttention | Efficient KV Cache Compression for Long Reasoning — Weian Mao Project Page
- TriAttention Compresses KV Cache 10.7x — danilchenko.dev
