Editorial illustration for NVIDIA GB300 NVL72 Achieves Record MoE Pre-Training Performance
NVIDIA GB300 Sets MoE Training Speed Record
NVIDIA GB300 NVL72 Achieves Record MoE Pre-Training Performance
NVIDIA's GB300 NVL72 system trained DeepSeek-V3, a 671-billion-parameter model, at 1,648 TFLOPs per GPU, a new record for mixture-of-experts pre-training. The number matters because it comes from a model class that has taken over frontier AI development for a specific reason: efficiency. DeepSeek-V3 carries 671 billion parameters total but activates only about 37 billion of them per token, so it runs at the compute cost of a much smaller dense model while still operating at frontier scale.
That efficiency comes with a cost of its own. Dense models activate every parameter for every token, which is expensive but architecturally simple. MoE models scatter their experts across GPUs, which means every layer has to route tokens out to the right experts and pull the results back, in both the forward and backward pass, through an all-to-all communication step.
That step sits directly in the critical path of training, repeated at every layer, every step, across thousands of GPUs. Small delays in that exchange stack up fast, and once communication can't be hidden behind compute anymore, throughput stalls even if more GPUs get added to the job.
Pre-training DeepSeek-v3 671B with 256 GPUs achieves a world record 1,648 TFLOPs per GPU with GB300 NVL72, enabling the same training job to reach the same performance with a fraction of the hardware compared to the previous generation.
Why this matters
The 1,648 TFLOPs-per-GPU figure on 256 chips matters less as a trophy and more as a signal of where the bottleneck has moved. NVIDIA's own framing is telling: with MoE cutting the compute per token, communication between GPUs becomes the thing that decides whether a cluster of thousands actually scales or just burns power. That's a systems problem, not a chip problem, which is why the story here is really about Megatron Core, TorchTitan, and JAX all getting tuned to run at full speed on GB300 NVL72, not just the silicon underneath them.
For developers and researchers, the practical takeaway is that benchmark numbers like this are only meaningful in context of the exact model, GPU count, and framework used, DeepSeek-V3 671B on 256 GPUs here, so treat comparisons across setups with caution. For founders sizing training budgets, the bigger signal is that NVIDIA is investing in open source training stacks directly, which suggests future performance gains will come as much from software plumbing as from new hardware generations. Worth watching whether these gains hold at larger GPU counts.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
Related Topic NVIDIA and Google Cloud let developers scale AI from prototype to production
Market Trends
Related Topic NVIDIA NeMo powers telco reasoning model for autonomous network workflows
Industry Applications
From Archives Xiaomi's New Robot AI Model Favors More Data Over Bigger Architectures
From Archives MCP Protocol Update Simplifies AI Server Session Management
From Archives NVIDIA's Cosmos 3 Edge Maps Robot Actions Into Compact Geometric Vectors
Open Source
From Archives NVIDIA's Sixth-Gen NVLink Powers Millions of AI Chips
Industry Applications
Common Questions Answered
What is the mixture-of-experts (MoE) efficiency advantage demonstrated by DeepSeek-V3?
DeepSeek-V3 contains 671 billion total parameters but only activates approximately 37 billion of them per token, allowing it to operate at frontier scale while consuming the computational resources of a much smaller dense model. This efficiency breakthrough is why MoE models have become dominant in frontier AI development.
What performance record did NVIDIA's GB300 NVL72 achieve with DeepSeek-V3 pre-training?
The GB300 NVL72 system achieved 1,648 TFLOPs per GPU when pre-training DeepSeek-V3 using 256 GPUs, setting a new world record for mixture-of-experts pre-training. This performance enables the same training job to reach equivalent results with significantly less hardware compared to previous generation systems.
Why does GPU-to-GPU communication matter more than individual chip performance for MoE pre-training?
With MoE models reducing compute requirements per token, communication between GPUs has become the primary bottleneck determining whether large GPU clusters can scale effectively or simply waste power. This shift means the performance challenge is now a systems-level problem requiring optimization of frameworks like Megatron Core, TorchTitan, and JAX rather than just improving individual chip capabilities.
How does the GB300 NVL72's efficiency compare to previous generation hardware for the same training task?
The GB300 NVL72 enables the same DeepSeek-V3 pre-training job to reach equivalent performance with a fraction of the hardware required by previous generation systems. This improvement in hardware efficiency represents a significant advancement in making frontier-scale AI model training more practical and cost-effective.
Further Reading
- NVIDIA Blackwell Tops MLPerf Training 6.0 with Industry-Leading Scale and Performance - NVIDIA Developer Blog
- NVIDIA GB300 Tops All Seven MLPerf 6.0 Benchmarks - AI Weekly
- FREE: NVIDIA Trains 671B-Parameter MoE Model Across 8,192 GPUs - Business Analytics Substack
- NVIDIA Blackwell Sweeps Every MLPerf 6.0 Benchmark With No Competition In Sight, While GB300 Systems Run Up to 60% Faster Than GB200 - WCCF Tech
- CoreWeave Trains DeepSeek-V3 Benchmark in Two Minutes - CoreWeave
