Editorial illustration for Nunchux AI Debuts VC-Attention, a Training-Free Kernel for Faster Video DiTs
Nunchux AI's VC-Attention Speeds Up Video Generation
Nunchux AI Debuts VC-Attention, a Training-Free Kernel for Faster Video DiTs
Attention is eating the clock on video generation. A 5-second clip at 720p from Wan2.2-14B breaks down into roughly 70,000 spatiotemporal tokens, and running full self-attention across all of them on an RTX 5090 now consumes more than 64% of total generation time. The pattern holds at the high end too: researchers behind Nunchux AI's new kernel say attention accounts for about two-thirds of every denoising step in MiniMax-H3 on a single B200 GPU. That's the problem Nunchux AI built VC-Attention to solve, a training-free, low-bit attention kernel aimed squarely at video Diffusion Transformers.
The fix isn't as simple as dropping precision on the matrix math. Low-bit Tensor Cores can speed up the QK and PV products, but two things get in the way. Existing smoothing methods like SageAttention2 handle queries and keys well, yet leave the value term responsible for 82% of output error on Wan2.2. Meanwhile the softmax step between those two products still runs in FP32, and on B200 and H200 hardware, that exponential calculation and its FP8 cast has become the slowest part of the whole pipeline.
Nunchux AI has released VC-Attention, a training-free low-bit attention kernel built for video Diffusion Transformers (DiTs). It targets 2 problems at once: value quantization error and a slow softmax stage.
Why this matters
Attention eating 64% of generation time on an RTX 5090 is the kind of number that explains why video generation still feels slow even on top-tier hardware. If VC-Attention's 6.02x speedup over SageAttention2 on B200 holds up outside Nunchux's own benchmarks, that's a real shift for anyone building or serving video DiTs, not just a paper exercise. The training-free part matters too: teams running Wan2.2-14B, HunyuanVideo-1.5, or similar models don't want to retrain or fine-tune just to get a faster kernel, and a drop-in low-bit attention path that preserves fidelity against BF16 FlashAttention-4 is the kind of thing infra teams will actually adopt.
We'd want to see independent benchmarks across more hardware than B200 and RTX 5090, and some clarity on how it degrades (or doesn't) at longer sequence lengths beyond the 70K-token clips tested here. Kernel-level tricks like this rarely make headlines, but they're often what determines whether a video model ships as a product or stays a demo. Worth watching whether Nunchux open-sources the kernel itself, not just the benchmarks.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Anthropic Merges Claude Chat and Cowork Into Single Product
From Archives FreezeVLA Study Shows One Adversarial Image Can Freeze a Robot
Common Questions Answered
What specific problems does VC-Attention solve in video Diffusion Transformers?
VC-Attention addresses two main issues: value quantization error and slow softmax computation stages in video DiTs. By tackling these bottlenecks simultaneously, it reduces the computational overhead that has been consuming over 60% of generation time in video generation models like Wan2.2-14B and MiniMax-H3.
Why is attention consuming so much processing time in video generation models?
Video generation models process enormous amounts of spatiotemporal tokens—a 5-second 720p clip from Wan2.2-14B breaks down into roughly 70,000 tokens. Running full self-attention across all these tokens requires substantial computation, accounting for approximately 64% of total generation time on high-end GPUs like the RTX 5090 and about two-thirds of each denoising step on a B200 GPU.
What is the performance improvement offered by VC-Attention compared to existing solutions?
VC-Attention delivers a 6.02x speedup over SageAttention2 on B200 GPUs according to Nunchux AI's benchmarks. This significant performance gain could substantially reduce video generation time for teams running popular models without requiring any retraining or fine-tuning of existing weights.
Why is the training-free nature of VC-Attention important for video generation teams?
The training-free design means teams running existing video generation models like Wan2.2-14B, HunyuanVideo-1.5, and similar architectures can immediately benefit from VC-Attention's speedups without having to retrain or fine-tune their models. This eliminates significant implementation barriers and allows for quick deployment across different video DiT architectures.
How does VC-Attention use low-bit quantization to improve video generation speed?
VC-Attention uses low-bit quantization on attention values while specifically addressing the quantization errors this typically introduces. By combining value quantization with optimizations to the softmax computation stage, it achieves faster processing without the accuracy degradation that usually accompanies aggressive quantization in attention mechanisms.
Further Reading
- Value Smoothing and Softmax Casting for Low-bit Attention - arXiv
- MonarchRT: Efficient Attention for Real-Time Video Generation - arXiv
- FPSAttention - ZIP Lab
- SLA: Beyond Sparsity in Diffusion ... - arXiv
- Sol-Attn: Accelerating Video Generation ... - Hugging Face Papers
