Editorial illustration for Moonshot AI releases FlashKDA with CUTLASS kernels, batching, H2O benchmarks
Moonshot AI releases FlashKDA with CUTLASS kernels,...
The race to scale long-context AI has a new contender. Moonshot AI just dropped FlashKDA, a production-grade CUDA kernel that turns their Kimi Delta Attention (KDA) from theory into hardware reality. KDA itself is no ordinary linear attention: it refines Gated DeltaNet with a channel-wise gating mechanism, squeezing more out of finite-state RNN memory.
But the real story is in the numbers. Kimi Linear, a 48B-parameter hybrid model with only 3B activated, pairs three KDA layers with every one global attention layer. That 3:1 ratio slashes KV cache by up to 75% during long sequences and unlocks 6× higher decoding throughput at a million-token context.
FlashKDA is the engine that makes the prefill fly, built on CUTLASS, optimized for variable-length batching, and benchmarked against H2O. This isn’t a research toy; it’s the core attention of a shipping model, open-sourced for anyone to use.
Kimi Delta Attention (KDA) is Moonshot AI’s contribution to this space: a linear attention mechanism that refines the Gated DeltaNet with a finer-grained, channel-wise gating mechanism, enabling more effective use of limited finite-state RNN memory. KDA is not just a research prototype. It is the core attention mechanism in Kimi Linear, Moonshot AI’s open-source hybrid model with 48B total parameters and 3B activated parameters.
Kimi Linear uses a 3:1 KDA-to-MLA (Multi-Head Latent Attention) ratio -- three KDA layers for every one global attention layer -- which reduces KV cache usage by up to 75% during long-sequence generation while achieving up to 6× higher decoding throughput at 1 million context length compared to full attention. FlashKDA is the production-grade CUDA kernel that makes that architecture fast during prefill. Concretely, the KDA forward pass takes in queries (
q), keys (k), values (v), a gate before activation (g), and beta logits (beta), along with ascalefactor, an output tensor (out), and gate parameters:A_log(log-gate parameter per head),dt_bias(gate bias), andlower_bound(gate lower bound, ranging from -5.0 to 0).
This is not a paper announcement. It is a working system dropped into the open, kernels and all, for anyone to run. FlashKDA proves that linear attention can be more than a theoretical curiosity.
When the gating is granular, channel-wise, not head-wise, the finite-state memory stops being a bottleneck and becomes an engine. At 48B parameters with only 3B activated, Kimi Linear already runs circles around full-attention baselines at million-token contexts. The 75% KV cache reduction is real.
The 6× decoding throughput is real. And now the prefill kernel is public. The benchmarks on H20 speak for themselves.
Variable-length batching, CUTLASS optimizations, raw performance that matches the promise. Moonshot AI has drawn the line between research and deployment, and they have put the code on the right side of that line. The message is clear: efficient long-context inference is no longer a dream.
It is a kernel you can compile today.
Related Reading
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
Open Source OpenAI's GPT Transcribe Cuts Error Rate to 3.31%, Improving on GPT-4o
Open Source Study: 6.7% of Deepfake Requests on Hugging Face Targeted Children
From Archives AI engineers abandon LangChain for native agents amid hidden complexity
From Archives Chinese AI startups drop offshore registration amid regulator IPO warning
Common Questions Answered
What is Kimi Delta Attention (KDA) and how does it improve upon GatedDeltaNet?
Kimi Delta Attention is a refined version of GatedDeltaNet that incorporates a channel-wise gating mechanism to enhance performance. This channel-wise approach, rather than head-wise gating, allows KDA to squeeze more capability out of finite-state RNN memory, making it more efficient than its predecessor.
How does FlashKDA enable production-grade deployment of linear attention?
FlashKDA is a production-grade CUDA kernel that transforms Kimi Delta Attention from theoretical research into practical hardware implementation. By providing optimized kernels, batching support, and H2O benchmarks, FlashKDA allows developers to deploy linear attention models in real-world applications rather than just academic settings.
What are the key performance metrics of Kimi Linear compared to full-attention baselines?
Kimi Linear is a 48B-parameter hybrid model with only 3B parameters activated, achieving a 75% KV cache reduction compared to full-attention models. At million-token contexts, Kimi Linear significantly outperforms full-attention baselines while maintaining this dramatic reduction in memory overhead.
Why is channel-wise gating more effective than head-wise gating in KDA?
Channel-wise gating in KDA provides more granular control over the finite-state memory mechanism compared to head-wise gating. This finer-grained approach prevents finite-state memory from becoming a bottleneck and instead transforms it into an engine that drives improved model performance.
Is FlashKDA available for public use or is it just a research announcement?
FlashKDA is not merely a paper announcement but a working system that has been released openly with kernels included for anyone to run. This open release allows the community to immediately implement and experiment with the linear attention technology in their own applications.
