Editorial illustration for DeepSeek AI unveils DeepSeek‑V4 with compressed attention for 1 M‑token contexts
DeepSeek-V4: 1M Token AI Model Redefines Context Limits
DeepSeek AI unveils DeepSeek‑V4 with compressed attention for 1 M‑token contexts
The AI arms race has always been a battle of scale, until now. DeepSeek’s latest release, DeepSeek-V4, doesn’t just push context windows to a million tokens; it fundamentally rewrites the efficiency equation. The secret lies in two architectural shifts.
First, Heavily Compressed Attention (HCA) discards the delicate dance of sparse selection entirely, instead merging every *m′* tokens into a single compressed entry and applying dense attention on those compacted representations. The result is brutal elegance: DeepSeek-V4-Pro slashes inference FLOPs to 27% of its predecessor and shrinks KV cache to a mere 10%. The Flash variant goes even further, 10% FLOPs, 7% cache.
But compression alone isn’t the whole story. DeepSeek-V4 replaces conventional residual connections with Manifold-Constrained Hyper-Connections (mHC), a structural rethinking that keeps gradients and representations stable across that million-token horizon. This isn’t an incremental step; it’s a deliberate dismantling of the old trade-offs between context length, compute, and memory.
HCA is more aggressive: it consolidates KV entries of every m′ tokens -- where m′ ≫ m into a single compressed entry, then applies dense attention over those representations. No sparse selection step is needed; the compression ratio itself reduces KV cache size. In the one-million-token setting, DeepSeek-V4-Pro requires only 27% of the single-token inference FLOPs (in equivalent FP8 FLOPs) and 10% of the KV cache size of DeepSeek-V3.2.
DeepSeek-V4-Flash achieves 10% of single-token FLOPs and 7% of KV cache relative to DeepSeek-V3.2. Manifold-Constrained Hyper-Connections (mHC) DeepSeek-V4 replaces conventional residual connections with Manifold-Constrained Hyper-Connections (mHC).
DeepSeek-V4 is not just a bigger context window. It is a rethinking of efficiency itself. By compressing attention where it matters, and removing the overhead of sparse selection, it turns a million tokens from a theoretical limit into a practical reality.
The numbers are stark: a fraction of the FLOPs, a sliver of the cache. Manifold-Constrained Hyper-Connections rewrite the rules of signal flow, replacing a tired architecture with something purpose-built for scale. This isn’t an incremental step.
It’s a new floor for what long-context models can achieve, and a clear signal that the era of brute-force scaling is giving way to something smarter.
Related Reading
Open Source Trump cracks down on Anthropic after Amazon tip; staff largely foreign
Open Source SDOF Adds Two Defensive Layers via Intent Router and StateAwareDisp
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
From Archives DeepSeek‑V4‑Pro‑Max tops open models, nears closed results at 1/6 Opus 4.7 cost
From Archives Update: Usage Limits Draining Faster Linked to Two Unrelated Experiments
Common Questions Answered
How does DeepSeek-V4 achieve a one-million-token context window?
DeepSeek-V4 uses two innovative attention techniques: compressed sparse attention and heavily compressed attention (HCA). HCA consolidates key-value entries of every m' tokens into a single compressed entry, dramatically reducing the key-value cache size and enabling much longer context windows.
What are the performance differences between DeepSeek-V4-Pro and DeepSeek-V4-Flash?
DeepSeek-V4-Pro has 1.6 trillion total parameters with approximately 49 billion activated per token, while DeepSeek-V4-Flash has 284 billion total parameters with 13 billion activations per token. Both models support a native one-million-token context, but differ significantly in scale and computational requirements.
What problem does Heavily Compressed Attention (HCA) solve in language models?
Heavily Compressed Attention (HCA) addresses the quadratic growth of key-value cache that traditionally limits context length in language models. By consolidating token entries and reducing cache size, HCA enables models like DeepSeek-V4 to achieve dramatically longer context windows with minimal computational overhead.
