Editorial illustration for MI355X FP8 vs. BF16 Kernel Speed Tested on Expert Sizes
MI355X FP8 vs BF16: MoE Kernel Speed Tested
AMD's Primus team just published numbers on how FP8 and BF16 kernels perform across different expert sizes on MI355X GPUs, and the results matter for anyone training mixture-of-experts models at scale. The blog is a follow-up to an earlier post on MoE training best practices, and this time the focus shifts to what's been built on top of that foundation: low-precision expert GEMMs in FP8 and MXFP8, a fused megakernel called MegaMoE that folds all-to-all communication into grouped GEMM operations, and training runs for DeepSeek-V3 spanning 256 to 1024 GPUs, including a 1700-step run with no interruptions. There's also a look at Primus Projection for Megatron-LM capacity planning and a JAX-based training path through MaxText.
All of it runs through Primus, AMD's framework for large-scale foundation model training and inference, which pairs with Primus-Turbo for high-performance operators and Primus-SaFE for platform stability. The work covers both the Megatron-LM and JAX backends, and it lands at a moment when open-source MoE architectures are multiplying in both size and design, forcing training infrastructure to catch up.
Low precision is the most direct lever on MoE training throughput, because expert GEMMs dominate compute. Primus supports FP8 training through Transformer Engine recipes with a Primus-Turbo operator overlay, covering delayed, tensorwise (current) scaling, blockwise scaling, and MXFP8 recipes.
Why this matters
The numbers in Figure 2 matter less for AMD's marketing than for anyone actually budgeting GPU-hours on MI355X clusters. Quantization overhead baked into the FP8 timing is the honest part: a lot of FP8-versus-BF16 comparisons quietly exclude that cost, which makes the kernel-level speedup here more trustworthy than most. Teams running MoE models at scale should care because grouped-GEMM is where tokens-per-expert actually gets processed, so a real win at the kernel level, not just at some synthetic benchmark, tends to show up in wall-clock training time rather than disappear into noise.
We'd still want to see this sweep repeated across more expert-size distributions before treating it as a general rule for every MoE architecture. Speedups that hold at one range of M don't automatically hold at another, and AMD's own framing (Primus, building on their prior MoE post) suggests this is one data point in an ongoing optimization effort, not a finished story. Worth watching: whether these kernel gains translate into end-to-end training throughput claims in a follow-up post, and whether NVIDIA's FP8 paths on comparable hardware get the same scrutiny.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google to End Free Access to Gemini AI for Non-Subscribers
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives China protests US Deepseek ban while blocking ChatGPT, Claude, Gemini at home
From Archives Claude Science Designs Scorpion Venom Protein Binders After 24-Hour Run
Common Questions Answered
What is the MegaMoE fused megakernel and how does it improve MoE training performance?
MegaMoE is a fused megakernel that folds all-to-all communication into grouped GEMM operations, optimizing the computational flow for mixture-of-experts models. By integrating communication directly into the grouped GEMM where tokens-per-expert are processed, it achieves real kernel-level speedups rather than just theoretical improvements.
How does FP8 precision training affect throughput in mixture-of-experts models on MI355X?
Low precision is the most direct lever on MoE training throughput because expert GEMMs dominate compute, making FP8 training particularly impactful. Primus supports FP8 training through Transformer Engine recipes with multiple scaling options including delayed, tensorwise, blockwise scaling, and MXFP8 recipes to maximize performance gains.
Why is the quantization overhead important when comparing FP8 versus BF16 kernel performance?
Most FP8-versus-BF16 comparisons quietly exclude quantization overhead costs, making them less trustworthy. The AMD Primus blog bakes quantization overhead into the FP8 timing measurements, providing more honest and reliable performance comparisons for teams budgeting GPU-hours on MI355X clusters.
What scaling options does Primus support for FP8 training in mixture-of-experts models?
Primus supports multiple FP8 scaling recipes including delayed scaling, tensorwise (current) scaling, blockwise scaling, and MXFP8 recipes. These different scaling approaches allow teams to optimize FP8 training based on their specific model architecture and performance requirements.
Further Reading
- Win on TCO: How AMD Instinct MI355X GPUs Achieve Cost Advantages for Large-Scale AI - AMD
- AMD Instinct MI355X GPUs - AMD
- AMD Instinct MI355X Drives Competitive Performance, Ideal for Optimized LLM Workloads - TechPowerUp
- DeepSeekV4 1.6T Day 0 to Day 43 Performance Over Time - GB300 NVL72, Huawei, MI355X, B200 - SemiAnalysis
- Primus-Turbo Mega MoE Documentation - AMD-AGI
