NVIDIA Nemotron Doubles Server Throughput with MoE

Why this matters

Nemotron-Labs-3-Puzzle-75B-A9B is a useful data point for anyone budgeting GPU fleets rather than benchmark scores. Cutting active parameters from 12.8B to 9B while holding user throughput steady and gaining 2.03x server throughput tells us NVIDIA is optimizing for the metric that actually shows up on a cloud bill: how many concurrent users a single H100 node can serve at a given per-user token rate. The FP8 weights, FP8 KV cache, and FP32 Mamba state details matter here too, since memory, not compute, is what caps context length at 1M tokens on one H100 (70GB of 80GB HBM already gone to weights alone).

For teams building on hybrid MoE architectures, this is a reminder that "compressed variant of a larger model" is becoming its own product category, not just a quantization afterthought. If you're serving Nemotron-3-Super in production, the real question is whether Puzzle-75B-A9B's accuracy holds up closely enough to justify the throughput gain, and NVIDIA's own paper is the place to check that math before you swap models. Watch for third-party benchmarks confirming the accuracy trade-off.