Editorial illustration for Microsoft Azure ND GB300 VM Boosts AI Performance with 50% More GPU Memory
Azure ND GB300 VM Boosts AI Compute by 50%
Microsoft's Azure ND GB300 VM hits 1.1 M tokens/sec, 50% more GPU memory
Microsoft's new server rack is fast. The Azure ND GB300 VM can process 1.1 million AI tokens per second, a number so large it feels like a physics problem, not a product spec.
Cloud providers keep tossing bigger hardware at AI. This one has 50% more GPU memory and a higher thermal design power, basic brute force upgrades for companies running massive models. The pitch is simple: rent this, and your inferences finish sooner.
The VM is optimised for inference workloads, featuring 50% more GPU memory and a 16% higher TDP (Thermal Design Power). To simulate the performance gains, Microsoft ran the Llama2 70B (in FP4 precision) from MLPerf Inference v5.1 on each of the 18 ND GB300 v6 virtual machines on one NVIDIA GB300 NVL72 domain. This used the NVIDIA TensorRT-LLM as the inference engine.
"One NVL72 rack of Azure ND GB300 v6 achieved an aggregated 1,100,000 tokens/s," said Microsoft. "This is a new record in AI inference, beating our own previous record of 865,000 tokens/s on one NVIDIA GB200 NVL72 rack with the ND GB200 v6 VMs." Since the system contains 72 Blackwell Ultra GPUs, the performance roughly translates to ~15,200 tokens/sec/GPU.
That speed, over fifteen thousand tokens per second for each of the 72 GPUs, is a record. It beats Microsoft's own previous best by a clear margin. The test used a specific, large model under ideal conditions. Real-world traffic is messier.
These benchmarks are marketing. They are also the only concrete measure we have. The numbers show a trajectory where the cost of running a complex AI query keeps falling, pushed by better hardware and software like NVIDIA's TensorRT-LLM.
The practical effect is that applications once deemed too slow or expensive become plausible. The boring infrastructure race, it turns out, defines what's possible.
Further Reading
Related Reading
Browse all Microsoft news →
Business & Startups Nordic pilot adds Gemini for Education, NotebookLM to boost AI literacy
Business & Startups Kling launches Video O1, all-in-one model with MVL bridge using transformer
Business & Startups DeepSeek Seeks More Capital Weeks After USD 7B Funding Round
Related Topic Claude gains shared context in Excel, PowerPoint; Microsoft adds Copilot Cowork
AI Tools & Apps
Related Topic Windows Copilot AI unable to pinpoint image source in user test
AI Tools & Apps
From Archives Infosys unveils Topaz Fabric, modular AI stack for IT ops, security
From Archives OpenAI, Microsoft call AGI pointless but tie it to billion-dollar deals
From Archives Meta, Google, Microsoft boost AI spending; capex hits USD 34.9B, up USD 5B
Market Trends
From Archives Microsoft Agent Lightning uses reinforcement learning to automate AI agent tuning
AI Tools & Apps
Common Questions Answered
How much GPU memory does the Azure ND GB300 VM offer compared to previous models?
The Azure ND GB300 VM provides 50% more GPU memory than previous virtual machine configurations. This significant memory increase is designed to support more complex machine learning and AI inference workloads, enabling faster and more efficient processing of large AI models.
What performance benchmark did Microsoft use to demonstrate the Azure ND GB300 VM's capabilities?
Microsoft used the Llama2 70B model in FP4 precision from MLPerf Inference v5.1 to benchmark the VM's performance. By running the test across 18 ND GB300 v6 virtual machines on a single NVIDIA GB300 NVL72 domain, they achieved an impressive 1.1 million tokens per second, setting a new record in AI inference performance.
What makes the Azure ND GB300 VM particularly suitable for AI and machine learning workloads?
The Azure ND GB300 VM is specifically optimized for inference workloads, featuring 50% more GPU memory and a 16% higher Thermal Design Power (TDP). These enhancements allow organizations to process complex machine learning models more efficiently, addressing the rapidly growing computational demands of AI infrastructure.
Further Reading
- Breaking the Million-Token Barrier - Azure ND GB300 v6 Virtual Machines with NVIDIA GB300 NVL72 rack-scale systems achieve unprecedented performance of 1,100,000 tokens/s on Llama2 70B Inference — Microsoft Tech Community
- Reimagining AI at scale: NVIDIA GB300 NVL72 on Azure — Microsoft Tech Community
- Building the future together: Microsoft and NVIDIA announce AI advancements at GTC DC — Azure Blog
- Microsoft Azure upgraded to NVIDIA GB300 'Blackwell Ultra' with 4600 GPUs connected together — TweakTown
- Nvidia and Microsoft to Redefine Data Centre Supercomputers — Data Centre Magazine
