Editorial illustration for Meta Tests MetaRoCE on 64-Node AMD Cluster, Outperforms RoCEv2
Meta's MetaRoCE Outperforms RoCEv2 on AMD Cluster
Meta ran a test this week that says a lot about where AI infrastructure bottlenecks actually live. On a 64-node cluster built with AMD Pensando programmable NICs, the company tried out MetaRoCE, a clean-sheet RDMA transport protocol designed from scratch for AI workloads running on ordinary Ethernet. The pitch is narrow but consequential: standard RoCEv2 assumes the network delivers frames in order, which means leaning on priority flow control and avoiding the packet spraying that actually helps performance at scale. MetaRoCE flips that assumption, treating the fabric as lossy by default and pushing ordering, path selection, and recovery down into the NIC itself.
This matters because training frontier models has become as much a networking exercise as a compute one. Collective operations like all-reduce and all-to-all force thousands of accelerators to sync up, and any straggler transfer stalls the whole job, stranding expensive GPU capacity in the process. Meta says it plans to release the MetaRoCE specification, a DPDK-optimized reference implementation, and a compliance test suite through the Open Compute Project, with a possible showing at the 2026 OCP Global Summit in October.
The design breaks with standard RoCE on its central assumption. Standard RoCE expects the network to deliver every frame in order, leveraging PFC and discouraging the packet spraying that provides performance in multiplane and large-scale networks. MetaRoCE instead treats the fabric as lossy and pushes ordering, path selection, and recovery into the NIC.
Why this matters
Meta building its own RDMA transport instead of tuning RoCEv2 tells us the hyperscalers no longer trust off-the-shelf networking stacks to keep pace with GPU clusters. The 64-node AMD test is small by Meta's own standards, but the choice of AMD Pensando NICs over Nvidia gear is worth noting: it signals Meta wants transport-layer control that isn't locked to one vendor's silicon. For researchers and infra teams, the real story is the resilience claim, not just throughput.
Flow completion times matter when you're paying for idle GPU-hours during an all-reduce stall, and if MetaRoCE genuinely handles failures better than RoCEv2, that's a cost line item, not a benchmark footnote. Founders building training infrastructure should watch whether Meta open-sources this or keeps it internal like much of its networking work. A protocol that only runs on Pensando NICs helps Meta's own fleet more than the broader ecosystem.
The bigger signal: network engineering is now a competitive axis for frontier labs, and whoever owns the transport layer owns a chunk of the economics of scale.
Related Reading
Browse all Meta news →
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
Related Topic WhatsApp launches Meta AI Incognito Chat, cuts latency for privacy
AI Tools & Apps
Related Topic Nvidia and Meta ink deal; Nvidia touts hardware for inference and AI training
Business & Startups
From Archives White House Slaps Export Controls on Anthropic After 24-Hour Whirlwind
From Archives Accel-Backed Keenable Builds Web Index for AI Agents
From Archives Meta AI Glasses Sales Surge Raises Covert Recording Concerns
Policy & Regulation
From Archives Meta Pays Microsoft Hundreds of Millions Annually for AI
Research & Benchmarks
Common Questions Answered
How does MetaRoCE differ from standard RoCEv2 in its approach to network packet delivery?
MetaRoCE treats the network fabric as lossy and pushes ordering, path selection, and recovery into the NIC itself, whereas standard RoCEv2 assumes the network delivers frames in order and relies on priority flow control. This fundamental design difference allows MetaRoCE to enable packet spraying across multiplane networks, which actually improves performance in large-scale deployments rather than limiting it as RoCEv2 does.
What hardware configuration did Meta use to test MetaRoCE on the 64-node cluster?
Meta built the 64-node test cluster using AMD Pensando programmable NICs rather than Nvidia networking equipment. This choice reflects Meta's desire to maintain transport-layer control that isn't locked to a single vendor's silicon, providing more flexibility for infrastructure teams.
Why did Meta decide to build its own RDMA transport protocol instead of optimizing RoCEv2?
Meta's development of MetaRoCE signals that hyperscalers no longer trust off-the-shelf networking stacks to keep pace with the demands of modern GPU clusters and AI workloads. The custom protocol addresses specific bottlenecks in AI infrastructure that standard protocols cannot adequately handle, particularly around packet spraying and network efficiency at scale.
What is the primary performance advantage of MetaRoCE over RoCEv2 for AI workloads?
MetaRoCE enables efficient packet spraying across multiplane and large-scale networks by treating the fabric as lossy and handling ordering and recovery at the NIC level. This approach provides better performance and resilience compared to RoCEv2's dependency on priority flow control and in-order frame delivery, which can become bottlenecks in hyperscale AI infrastructure.
Further Reading
- MetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet - Meta Engineering
- Meta's custom transport protocol embraces packet chaos to boost AI throughput - SDxCentral
- Meta has announced 'MetaRoCE,' a new communication protocol for large-scale AI infrastructure - GIGAZINE
- Comparative Analysis of Scale-Out RoCE Network Traffic Patterns - AMD ROCm Blog
- RoCE networks for distributed AI training at scale - Meta Engineering
