Editorial illustration for Matmul Enables Dropless MoE Training; Grouped‑GEMM Kernel Drives Speed
Matmul Enables Dropless MoE Training; Grouped‑GEMM...
Dropless Mixture-of-Experts training has long been haunted by a simple truth: you cannot skip the tokens, but you can pay for them in compute. The matmul is what makes dropless possible at all, but without a well-tuned grouped-GEMM kernel, that possibility remains academic. MegaBlocks proved this on GPUs.
Now DeepEP tackles the remaining bottleneck, the routing all-to-all. Its dispatch and combine kernels move tokens across ranks with token-aware precision, leaning on NVLink and RDMA. The receive buffer is still worst-case, but leaner chunked transfers shrink the transient footprint.
This is the architecture that turns dropless from a theory into a practical speed advantage.
You’ll learn how grouped GEMM and DeepEP work, how to integrate a custom kernel through JAX’s FFI — custom VJPs, sharding contracts, and a once-per-process bootstrap included — and how the dropless path stacks up against the capacity-factor default on both throughput and convergence. By the end, you’ll see how Primus-Turbo turns dropless MoE training on AMD Instinct GPUs from infeasible into a practical, faster, and more memory-efficient default.
The grouped-GEMM kernel tore down the matmul bottleneck, turning a theoretical advantage into a practical one. DeepEP then did the same for the all-to-all routing. It does not eliminate the worst-case buffer, physics and parallelism demand that, but it manages the transient footprint with surgical precision.
Chunked sends, fewer copies, a leaner pipeline. Dispatch and combine, mirror opposites, now complete a round-trip that is both token-aware and expert-efficient. This is the formula: a well-tuned primitive for compute, a purpose-built library for communication.
Together, they make dropless MoE training not just feasible, but fast. The wall has been moved.
Related Reading
Open Source Trump cracks down on Anthropic after Amazon tip; staff largely foreign
Open Source SDOF Adds Two Defensive Layers via Intent Router and StateAwareDisp
Open Source D&B rebuilds 642 million‑business database after AI agents hit limits
From Archives LangChain Emergency Helpline Uses AssemblyAI WebSocket for Live STT
From Archives Meta launches Hatch AI agent, its first paid product, priced up to USD 200/month
Common Questions Answered
What is the key innovation that makes dropless Mixture-of-Experts training practical?
The grouped-GEMM kernel is the critical innovation that enables dropless MoE training by eliminating the matmul bottleneck. While dropless MoE theoretically allows processing all tokens without skipping, the grouped-GEMM kernel transforms this theoretical advantage into practical performance by optimizing the matrix multiplication operations that would otherwise consume excessive compute resources.
How does DeepEP address the routing all-to-all bottleneck in MoE systems?
DeepEP tackles the all-to-all routing bottleneck through specialized dispatch and combine kernels that move tokens across ranks with token-aware precision. These kernels leverage NVLink and RDMA technologies to efficiently manage token routing while reducing transient buffer footprint through techniques like chunked sends and fewer copies.
What problem did MegaBlocks solve and what remaining challenge does DeepEP address?
MegaBlocks proved that the grouped-GEMM kernel could overcome the matmul bottleneck on GPUs, making dropless MoE training feasible. DeepEP then tackled the remaining bottleneck by optimizing the routing all-to-all communication, completing the efficiency improvements needed for practical dropless MoE training at scale.
Why can't dropless MoE training completely eliminate worst-case buffer requirements?
Physics and parallelism constraints inherently demand some worst-case buffer overhead that cannot be eliminated entirely. However, DeepEP manages the transient footprint with surgical precision through optimized dispatch and combine operations, minimizing buffer usage even though complete elimination remains impossible due to fundamental system constraints.
Further Reading
- Training MoEs at Scale with PyTorch — PyTorch Blog
- MegaBlocks: Efficient Sparse Training with Mixture-of-Experts — arXiv
- MoE Training Optimization — Megatron Bridge — NVIDIA Docs
- Mixture of experts with Dropless Computation — SugiV Blog
- Explore Mixture of Experts (MoE) inference support for Neuron — AWS Docs
