Editorial illustration for NVIDIA's NVFP4 Training Recipe Boosts AI Speed and Cuts Costs
NVFP4: 4-Bit AI Training Breakthrough Cuts Model Costs
NVIDIA's NVFP4 Training Recipe Boosts AI Speed and Cuts Costs
Nvidia's own hardware keeps looking obsolete—because Nvidia's software keeps making it so. Take training a giant AI model. Months ago, that required 512 of their top-tier Blackwell GPUs.
Now? A new software trick called NVFP4 gets the same job done nearly twice as fast. No new silicon.
Just a better recipe, using what's already in the data center, to slash costs by making everything run faster.
NVIDIA also recently published an NVFP4 training recipe, bringing the significant performance benefits of NVFP4 to model training, enabling model makers to train AI faster and at lower cost. In the latest version of the MLPerf Training benchmark suite, multiple NVIDIA GB300 NVL72 systems--totaling 512 Blackwell Ultra GPUs--worked together using NVFP4 precision to complete the Llama 3.1 405B pre-training benchmark in 64.6 minutes. This is 1.9x faster than 512 Blackwell GPUs across multiple NVIDIA GB200 NVL72 systems, which were able to complete the benchmark using FP8 in the prior round. Looking ahead, the NVIDIA Rubin platform delivers large leaps in NVFP4 capability for training and inference, offering 35 petaFLOPS of NVFP4 training compute, and 50 petaFLOPs of NVFP4 Transformer Engine inference compute.
The proof is in the MLPerf result: 64.6 minutes. That single number, for pre-training the Llama 3.1 405B model, translates directly to less money burned on power, cooling, and rack time. And Nvidia isn't stopping with a software win.
The next move is the Rubin platform, promising 35 petaFLOPS for NVFP4 training and 50 for inference. The industry scrapes for single-digit efficiency gains. Nvidia just delivered a 90% speed-up with code, then announced the hardware sledgehammer to back it up.
The goal is to make training a frontier model a manageable cost, not a prohibitive barrier.
Related Reading
Industry Applications Westinghouse teams with Google Cloud to build AI platform for nuclear power
Industry Applications NVIDIA NeMo powers telco reasoning model for autonomous network workflows
Industry Applications Month-1 Agent Adds Holistic Observability with Trace IDs and Token Tracking
Related Topic NVIDIA and Google Cloud let developers scale AI from prototype to production
Market Trends
Related Topic China offers cheaper electricity to AI firms abandoning NVIDIA chips
Business & Startups
From Archives Google launches Personal Intelligence in AI Mode for Pro and Ultra users
From Archives Google unveils Gemini and Classroom updates at BETT conference
From Archives Build Kimi K2.5 Multimodal VLM with NVIDIA GPU-Accelerated Endpoints
LLMs & Generative AI
From Archives Nvidia CEO says claim he's unhappy with OpenAI 'nonsense' and rejects USD 100B plan
Business & Startups
Common Questions Answered
How does NVIDIA's NVFP4 format improve large language model training efficiency?
[nvidia.com](https://developer.nvidia.com/blog/nvfp4-trains-with-precision-of-16-bit-and-speed-and-efficiency-of-4-bit/) reveals that NVFP4 enables 4-bit pretraining by cutting memory needs and boosting arithmetic throughput. The format uses techniques like micro-block scaling, high-precision block encoding, and stochastic rounding to maintain model accuracy during large-scale training, allowing AI factories to scale more rapidly and sustainably.
What was the key achievement in NVIDIA's NVFP4 training research?
[tomshardware.com](https://www.tomshardware.com/tech-industry/artificial-intelligence/nvidia-details-efficiency-of-the-nvfp4-format-for-llm-training-new-paper-reveals-how-nvfp4-offers-benefits-over-fp8-and-bf16) reports that NVIDIA successfully trained a 12-billion-parameter model on 10 trillion tokens using NVFP4, which is the longest publicly documented training run in 4-bit precision. The experiment demonstrated that NVFP4 could achieve accuracy comparable to higher precision formats like FP8, showcasing its potential for efficient large-scale model training.
What specific techniques did NVIDIA use to enable stable 4-bit model training?
[huggingface.co](https://huggingface.co/papers/2509.25149) highlights that NVIDIA's approach included Random Hadamard transforms to bound block-level outliers, a two-dimensional quantization scheme for consistent representations, stochastic rounding for unbiased gradient estimation, and selective high-precision layers. These innovative techniques allowed the researchers to overcome traditional challenges of training in 4-bit precision, such as maintaining training stability and convergence.
Further Reading
- NVFP4 Trains with Precision of 16-Bit and Speed and Efficiency of 4-Bit — NVIDIA Developer Blog
- How Nvidia solved the accuracy tradeoff of training 4-bit LLMs — BD Tech Talks
- Accelerating large language models with NVFP4 quantization — Red Hat Developer
