Editorial illustration for Google DeepMind's Decoupled DiLoCo hits 88% goodput despite hardware failures
DeepMind DiLoCo: AI Training Breakthrough at 88% Efficiency
Google DeepMind's Decoupled DiLoCo hits 88% goodput despite hardware failures
Google DeepMind’s latest paper unveils Decoupled DiLoCo, an asynchronous training framework that keeps more than eight‑in‑ten chips busy even when a sizable slice of the hardware drops out. The team measured a striking 88 % goodput while deliberately injecting failure rates that would cripple conventional pipelines. That performance matters because large‑scale model training today leans on data‑parallel schemes that assume a near‑perfect fabric of connectivity and hardware reliability.
When thousands of accelerators are spread across several data centers, any hiccup in the network or a single node’s crash can stall the whole job, turning cost and time into prohibitive factors. By decoupling computation from synchronization, DiLoCo promises to sidestep those roadblocks, offering a path forward for researchers who need to push models beyond the limits of current infrastructure.
Across thousands of chips spanning multiple data centers, that bottleneck is not just inconvenient; it makes global‑scale training effectively impractical. Conventional Data‑Parallel training requires approximately 198 Gbps of inter‑datacenter bandwidth across eight data centers — far beyond what s
Across thousands of chips spanning multiple data centers, that bottleneck is not just inconvenient; it makes global-scale training effectively impractical. Conventional Data-Parallel training requires approximately 198 Gbps of inter-datacenter bandwidth across eight data centers -- far beyond what standard wide-area networking (WAN) can support between geographically distributed facilities. How Decoupled DiLoCo Works Decoupled DiLoCo builds on two prior systems from Google.
The first is Pathways, which introduced a distributed AI system based on asynchronous data flow, allowing different compute resources to work at their own pace without blocking on one another. The second is DiLoCo, which dramatically reduced the inter-datacenter bandwidth required for distributed training by having each worker perform many local gradient steps before communicating with peers -- dramatically reducing how much data needs to flow between data centers. Built on top of Pathways, training is divided across separate clusters of accelerators called learner units -- the 'islands' of compute.
Each learner unit trains semi-independently, performing many local steps, before sharing a compressed gradient signal with an outer optimizer that aggregates updates across all learner units. Because this outer synchronization step is asynchronous, a chip failure or slow learner unit in one island does not block the others from continuing to train. Decoupled DiLoCo reduces required inter-datacenter bandwidth from 198 Gbps to just 0.84 Gbps across eight data centers -- multiple orders of magnitude lower -- making it compatible with standard internet-scale connectivity between datacenter facilities rather than requiring custom high-speed network infrastructure.
Self-Healing Through Chaos Engineering One of the most technically significant properties of Decoupled DiLoCo is its fault tolerance. The research team used chaos engineering, a method that deliberately introduces artificial hardware failures into a running system to test its robustness during training runs.
Will training at this scale become feasible? Decoupled DiLoCo shows an 88 % goodput even when many chips falter, suggesting an asynchronous approach can tolerate failure. The architecture separates gradient computation from synchronization, allowing thousands of processors across multiple data centers to keep moving while some stall.
Conventional data‑parallel schemes, by contrast, demand roughly 198 Gbps of inter‑datacenter bandwidth across eight sites—a requirement the article notes exceeds current capabilities. Yet the report stops short of quantifying overhead, energy impact, or how the system behaves under different failure patterns. Moreover, the excerpt ends abruptly, it's unclear whether the bandwidth figure applies to a specific workload or represents a theoretical ceiling.
The researchers’ claim that “global‑scale training becomes practical” hinges on assumptions not fully detailed. As the field pushes toward models with hundreds of billions of parameters, the trade‑offs between asynchrony and model convergence remain to be validated in broader settings. Until more data emerge, the practical limits of Decoupled DiLoCo are still uncertain.
Further Reading
- DeepMind's DiLoCo Revolutionizes Language Model Training with 500x Less Communication - Synced
- Google DeepMind Researchers Introduce DiLoCo: A Novel Distributed Low-Communication Machine Learning Algorithm for Effective and Resilient Large Language Model Training - MarkTechPost
- OpenDiLoCo: An Open-Source Framework for Globally Distributed Training - PrimeIntellect
- DeepMind working on distributed training of large AI models - The Register
Related Reading
Browse all Google news →
Research & Benchmarks Hyperparameter Tuning Reaches 0.9617 Accuracy in 64.59 Seconds
Research & Benchmarks Pharma Cautious as AI Promises Faster Drug Discovery and Smarter Trials
Research & Benchmarks Google AI Advisors Let Users Probe Performance with Conversational “Why” Queries
Related Topic Gemini 3 Pro builds screenshot-to-code app in two prompts, fixes bugs
LLMs & Generative AI
Related Topic Gemini 3 Pro and GPT-5 stumble on graduate-level physics benchmark
LLMs & Generative AI
From Archives Agent observability powers production evaluation through trace analysis
From Archives Xiaomi launches MiMo‑V2.5‑Pro and V2.5, matching benchmarks at lower token cost
From Archives Google Cloud AI launches ReasoningBank with MaTTS memory-aware scaling
Policy & Regulation
From Archives Google unifies Gemini Enterprise Platform and Application in new release
Business & Startups
Common Questions Answered
How does Decoupled DiLoCo achieve 88% goodput during large-scale machine learning training?
Decoupled DiLoCo uses an asynchronous training framework that separates gradient computation from synchronization, allowing processors to continue working even when some hardware components fail. The approach enables thousands of processors across multiple data centers to maintain productivity, unlike conventional data-parallel training methods that halt when hardware issues occur.
What bandwidth challenges does Decoupled DiLoCo address in distributed machine learning?
Conventional data-parallel training requires approximately 198 Gbps of inter-datacenter bandwidth across eight data centers, which exceeds current wide-area networking (WAN) capabilities. Decoupled DiLoCo overcomes this limitation by creating an architecture that can tolerate hardware failures and maintain training efficiency across geographically distributed facilities.
Why is the 88% goodput metric significant for machine learning training infrastructure?
The 88% goodput demonstrates that an asynchronous training approach can maintain high computational efficiency even when substantial hardware components are non-functional. This breakthrough suggests more resilient and scalable training methods for large machine learning models across distributed computing environments.