Editorial illustration for Zyphra AI Introduces ZHA, a Path to True Subquadratic Attention
Zyphra's ZHA Achieves True Subquadratic Attention
Zyphra AI Introduces ZHA, a Path to True Subquadratic Attention
Zyphra has put out a paper on Zyphra Hierarchical Attention, a sparse attention method that tries to fix a problem other sparse methods never actually solved. The pitch for sparse attention has always been speed: skip the tokens that don't matter, only compute attention on the ones that do. But something still has to decide which tokens matter, and that something is usually an indexer that scores every single token in the context.
DeepSeek Sparse Attention works this way. So the final attention step gets cheaper, but the indexer scanning the whole context doesn't, and at contexts approaching 1 million tokens that indexing cost starts to dominate the runtime anyway.
ZHA's answer is a tree-structured indexer that searches for relevant keys instead of scoring all of them, while keeping the full KV cache intact. Zyphra distilled it into a frozen Qwen3.6-35B-A3B model using roughly 4 billion tokens, adding a negligible number of trainable parameters, and tested it against dense attention across reasoning, agentic coding, and 256K-context retrieval benchmarks.
Zyphra has released Zyphra Hierarchical Attention (ZHA), a sparse attention method built around a tree-structured indexer. ZHA targets a known gap in sparse attention: the indexer that picks tokens still scans the whole context. Zyphra’s technical paper shows ZHA can be distilled into frozen pretrained models with about 4B tokens.
Why this matters
The indexer bottleneck in sparse attention is one of those problems that's easy to miss until someone shows you the numbers: 60 to 90% of layer time spent scoring tokens you're about to throw away. That's the real cost of context scaling past 500K tokens, and it's been hiding behind the headline sparsity gains. ZHA's tree-structured approach to keeping the indexer itself subquadratic is a more honest fix than most "sparse attention" releases, which tend to optimize the easy part and leave the expensive part alone.
What should catch the attention of people building long-context systems is the distillation path: 4B tokens to retrofit a frozen Qwen3.6-35B-A3B without wrecking reasoning or 256K retrieval scores. That's a retrofit cost most teams could actually afford, not a from-scratch pretrain. We'd want to see this tested on models outside Zyphra's own benchmarks before treating it as a general solution, and the paper's claims about near-dense performance deserve independent replication. But if the indexer problem is as real as the cited 60-90% figure suggests, this is worth watching closely.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives 36 Strong Simple Baselines Assembled From Nature, Science, Cell-family Studies
From Archives NVIDIA Vera Rubin NVL72 Kubernetes AI-Ready: Network Operator, DMA Topology
Common Questions Answered
What is the main limitation of existing sparse attention methods like DeepSeek Sparse Attention?
The primary bottleneck in existing sparse attention methods is that the indexer used to select which tokens matter still needs to scan and score every single token in the context. This means that despite the sparsity gains, 60 to 90% of layer computation time is spent scoring tokens that will ultimately be discarded, making the indexer itself a quadratic bottleneck.
How does Zyphra Hierarchical Attention (ZHA) solve the indexer bottleneck problem?
ZHA uses a tree-structured indexer design that keeps the indexer itself subquadratic, rather than just optimizing the attention computation after token selection. This architectural approach directly addresses the hidden cost of context scaling beyond 500K tokens by making the indexing process more efficient.
Can Zyphra Hierarchical Attention be integrated into existing pretrained models?
Yes, according to Zyphra's technical paper, ZHA can be distilled into frozen pretrained models using approximately 4 billion tokens. This means the method can be adapted to existing models without requiring complete retraining from scratch.
Why is ZHA considered a more honest fix compared to other sparse attention releases?
Most sparse attention methods optimize only the easy part of the problem by reducing computation in the attention mechanism itself, while ignoring the substantial indexer overhead. ZHA addresses the actual bottleneck by making the indexer subquadratic, providing a more comprehensive solution to the true cost of sparse attention at scale.
Further Reading
- Subquadratic Attention with a Hierarchical Tree-Structured Indexer - arXiv
- Zyphra AI Introduces Zyphra Hierarchical Attention (ZHA): A Path to True Subquadratic Attention - MarkTechPost
- Zyphra introduces hierarchical attention as a path to subquadratic scaling - Digg
- The Sparse Frontier: Sparse Attention Trade-offs in Transformer - ACL Anthology
- Benchmarking Subquadratic's latest model & SSA Kernel - Appen
