Editorial illustration for MRC retires paths, then probes to confirm failures and recovery
MRC retires paths, then probes to confirm failures and...
Packet loss plagues networks. Most guess at its cause. OpenAI's MRC system does not guess.
When it detects loss, it immediately takes a path out of service. Then it runs a test. It must confirm a real failure and monitor for recovery.
But loss isn't always a broken wire. Congestion at a destination server is a common culprit. To distinguish a traffic jam from a true road collapse, an MRC switch facing a full queue performs surgery.
It does not drop the packet. It cuts off the data payload and forwards only the header. This forces the receiver to ask for a retransmission directly.
That precise trim slashes false alarms.
After MRC retires a path, it sends probe packets to check whether there really was a failure, and if so, whether it has recovered. Failures aren't the only cause of packet loss though; another common source of loss is congestion at the destination. If a switch would otherwise drop a packet due to congestion, it trims off the payload and forwards only the header to the destination, triggering an explicit retransmission request.
Packet trimming reduces false positives where we incorrectly assume a loss means a path has failed. This combination of multi-plane topology, spraying, load-balancing, and trimming means that an MRC connection can detect network failures and route around them on a microsecond timescale, minimizing the impact on synchronous training jobs. In contrast, a conventional network fabric could take seconds or even tens of seconds to stabilize and route around failures.
MRC allows us to go one step further in simplifying our networks. Traditionally, switches run a dynamic routing protocol such as BGP (Border Gateway Protocol) to compute available paths and route around failures. But switches are complex devices running complex software.
When they fail in subtle ways, those problems can be hard to diagnose and can cause connection failures until fixed. If packets are lost on a path, MRC stops using that path. We took the more radical approach of disabling dynamic routing and using IPv6 Segment Routing (or SRv6), instead.
SRv6 lets the sender directly specify the path each packet should take through the network. It does this by embedding the sequence of switch identifiers into each packet's destination address. Breaking this down: When forwarding, a switch checks if its own identifier is present.
If it is, it removes the identifier by shifting the destination address so that the next switch's identifier is revealed.
The radical step was stripping out protocols like BGP from the switches. Entirely. The replacement is IPv6 Segment Routing.
Here, the sender encodes the full path—every switch hop—inside each packet’s destination address. A switch finds its own ID, strips it away, and reveals the next stop. This architectural simplification is critical.
Paired with failure detection measured in microseconds, not seconds, it allows massive training jobs to run with far fewer interruptions. A typical network fabric might stutter for ten seconds or more just to reroute. MRC does not wait.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives eOptShrinkQ enables near‑lossless KV cache compression with spectral denoising
From Archives Harvard study finds OpenAI's o1 and 4o outdiagnose ER doctors in 76‑patient test
Common Questions Answered
How does OpenAI's MRC system detect and respond to packet loss in networks?
When MRC detects packet loss, it immediately takes the affected path out of service and runs a test to confirm whether it's a real failure or temporary congestion. This rapid detection and isolation approach allows the system to distinguish between actual network failures and traffic jams at destination servers, enabling faster recovery times measured in microseconds rather than seconds.
What technique does MRC use to differentiate between congestion and actual network failures?
When an MRC switch faces a full queue, it performs surgery on the data by cutting off the packet payload while keeping the header intact, rather than dropping the entire packet. This allows the system to determine whether the issue is congestion at a destination server or a true network failure, enabling more accurate failure detection and response.
Why did OpenAI replace BGP with IPv6 Segment Routing in the MRC architecture?
OpenAI stripped out traditional protocols like BGP entirely and replaced them with IPv6 Segment Routing to simplify the network architecture. In this system, the sender encodes the full path—every switch hop—inside each packet's destination address, allowing switches to find their own ID, strip it away, and reveal the next stop, which is critical for achieving microsecond-level failure detection.
What is the key advantage of MRC's architectural simplification with IPv6 Segment Routing?
By encoding the complete path inside each packet's destination address rather than relying on complex routing protocols, MRC achieves massive scalability paired with failure detection measured in microseconds instead of seconds. This architectural simplification eliminates the overhead of traditional routing protocols while enabling rapid detection and recovery from network failures.
Further Reading
- Papers with Code Benchmarks — Papers with Code
- Chatbot Arena Leaderboard — LMSYS
