Editorial illustration for Prime Intellect Releases Trainable Mega MoE Kernel, 3.11x Faster Forward Pass
Prime Intellect Releases Trainable Mega MoE Kernel
Prime Intellect released Prime Mega MoE on its open-source stack, a kernel that fixes a gap DeepSeek left open. DeepSeek's Mega MoE kernel, built into the DeepGEMM library, only handled the forward pass. Training needs a backward pass too, and that's what Prime Intellect built: a fast, overlapping backward pass that turns DeepSeek's inference-only kernel into something usable for actual training runs.
The numbers come from testing on Qwen3-30B-A3B and Qwen3-235B-A22B, both running 128 experts with top-8 routing. On the prime-rl trainer, full training steps run 32% to 34% faster. Against TorchTitan, the gap widens to 48% to 55%. Tests ran on 8×H200 nodes, scaling from 32 to 128 GPUs, with support for both Hopper and Blackwell hardware.
The speedup matters because MoE training is bottlenecked by communication, not compute. Each token routes to a handful of experts scattered across GPUs, meaning every layer ships tokens out and pulls results back, twice per layer once you count the backward pass.
Prime Intellect has released Prime Mega MoE, a training extension of DeepSeek’s fused Mega MoE kernel. DeepSeek’s kernel only ran the forward pass. Prime Intellect added a fast, overlapping backward pass.
Why this matters
GPU hours are the real budget line in MoE training, and a 32-55% cut in step time against TorchTitan and the prime-rl baseline is the kind of number that changes procurement decisions, not just benchmark slides. What's notable here isn't the raw speedup but the move itself: Prime Intellect took a kernel DeepSeek shipped for inference only and built out the backward pass needed to make it trainable, then packaged it as a prebuilt DeepGEMM wheel through prime-kernels. That's a meaningfully lower barrier than asking every team to write their own fused MoE training kernel from scratch.
For researchers and infra teams working on Qwen3 and similar MoE architectures, this is worth testing directly against your own training stack rather than taking the percentages at face value, since gains against TorchTitan and gains against prime-rl aren't apples to apples. Still, the direction is clear: fused kernels are moving from inference-only tooling into core training infrastructure, and whoever controls that layer shapes how cheaply the next generation of MoE models actually gets trained.
Related Reading
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
Open Source Box CEO Aaron Levie Says AI Will Drive "Biggest Cybersecurity Upgrade
Open Source Study: 6.7% of Deepfake Requests on Hugging Face Targeted Children
From Archives Mistral releases weights for trillion-parameter AI, targeting cybersecurity
From Archives Mistral releases free Le Chonk, says it leapfrogged China in open-weight race
Common Questions Answered
What limitation did DeepSeek's Mega MoE kernel have that Prime Intellect addressed?
DeepSeek's Mega MoE kernel, built into the DeepGEMM library, only handled the forward pass and was designed for inference only. Prime Intellect solved this gap by implementing a fast, overlapping backward pass that enables the kernel to be used for actual training runs, making it a complete trainable solution.
How much faster is Prime Intellect's Mega MoE kernel compared to existing baselines?
Prime Intellect's Mega MoE kernel delivers a 3.11x faster forward pass and achieves a 32-55% reduction in step time compared to TorchTitan and the prime-rl baseline. These significant performance improvements directly impact GPU hour budgets and can influence procurement decisions for MoE training operations.
What models were used to test Prime Intellect's Mega MoE kernel performance?
The performance benchmarks for Prime Intellect's Mega MoE kernel were tested on Qwen3-30B-A3B and Qwen3-235B-A22B models, both running with 128 experts. These diverse model sizes demonstrate the kernel's effectiveness across different scales of Mixture of Experts architectures.
How is Prime Intellect's Mega MoE kernel distributed to users?
Prime Intellect has packaged Prime Mega MoE as a prebuilt DeepGEMM wheel through prime-kernels, making it easy for users to integrate into their training workflows. This distribution method allows developers to quickly adopt the optimized kernel without complex compilation or setup processes.
