Editorial illustration for Sakana AI's PC-ALM Trains 1000-Layer Networks Without Backpropagation
Sakana AI Trains 1000-Layer Networks Without Backprop
Sakana AI researchers have built a training method that skips backpropagation entirely and still scales to networks 1000 layers deep. The approach, called Augmented Lagrangian Predictive Coding (PC-ALM), tackles a problem that's dogged neuroscience-inspired learning for decades: backprop requires a forward pass and a backward pass to run in lockstep, with weight updates locked behind both. No one has found evidence that brains work this way.
Predictive coding offered an alternative, updating each layer using only local information, but it had a well-documented flaw. Innocenti et al. showed that standard predictive coding falls apart in deep, narrow networks, where supervision signals from the output layer have to pass through too many local compromises before reaching the earlier layers.
The signal degrades along the way, and the gap between predictive coding and backprop performance grows with depth.
Sakana AI's fix reframes training as a constrained optimization problem rather than a chain of local approximations. The team says PC-ALM trains residual MLPs at extreme depth while landing within about 2 percentage points of backprop accuracy on MNIST, using an MIT-licensed JAX implementation that runs on CPU.
Sakana AI researchers propose Augmented Lagrangian Predictive Coding (PC-ALM), a variant of PC that keeps every update layer-local yet recovers backprop-aligned credit signals. The research team reports training residual MLPs up to 1000 layers within about 2 percentage points of backprop on MNIST.
Why this matters
Backpropagation's phase-locking has always been the elephant in the room for anyone trying to scale training across exotic hardware or biologically-inspired architectures. Sakana AI's PC-ALM doesn't just tinker with predictive coding, it reframes the whole problem as constrained optimization and shows the constraint-satisfaction view can close the gap with backprop at depths (1000 layers) where standard PC falls apart. For researchers, the width-versus-depth characterization of the PC-BP gap is the more interesting result here: it gives a concrete diagnostic for when local learning rules will underperform, rather than a vague "sometimes it works" caveat.
For developers and founders eyeing distributed or neuromorphic training setups, this is worth watching precisely because layer-local updates remove the global synchronization bottleneck that makes backprop awkward on decentralized or asynchronous hardware. We'd temper enthusiasm until PC-ALM gets tested on tasks beyond whatever benchmarks Sakana used internally, and until someone independently verifies the depth claims. But a credit-assignment method that scales to four-digit layer counts without a backward pass is the kind of result that deserves scrutiny, not a shrug.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives AI Agents Report Fellow Agents for Cheating, Study Finds
From Archives Microsoft AI CEO calls Anthropic speculation 'really, really dangerous
Common Questions Answered
What is Augmented Lagrangian Predictive Coding (PC-ALM) and how does it differ from backpropagation?
PC-ALM is a training method developed by Sakana AI researchers that eliminates the need for backpropagation entirely by reframing the training problem as constrained optimization. Unlike backpropagation, which requires synchronized forward and backward passes with locked weight updates, PC-ALM keeps every update layer-local while still recovering credit signals aligned with backprop performance, making it more biologically plausible and hardware-efficient.
How deep can neural networks be trained using PC-ALM according to Sakana AI's research?
Sakana AI researchers successfully trained residual MLPs up to 1000 layers deep using PC-ALM, achieving performance within approximately 2 percentage points of standard backpropagation on MNIST. This represents a significant breakthrough, as previous predictive coding approaches struggled to scale effectively at such extreme depths.
Why is backpropagation's phase-locking considered problematic for biological and hardware-based learning?
Backpropagation requires forward and backward passes to run in lockstep with weight updates locked behind both operations, a constraint that has no evidence of occurring in biological brains and creates inefficiencies for exotic hardware architectures. This phase-locking limitation has motivated researchers to seek alternative training methods that can operate more independently across layers, which PC-ALM addresses through its layer-local update mechanism.
What problem has predictive coding traditionally faced that PC-ALM helps solve?
Traditional predictive coding approaches have struggled to scale effectively to very deep networks, falling apart at extreme depths where backpropagation remains reliable. PC-ALM overcomes this limitation by combining predictive coding with constrained optimization techniques, enabling stable training of networks with 1000 layers while maintaining the biological plausibility advantages of the predictive coding framework.
Further Reading
- Augmented Lagrangian Predictive Coding - arXiv
- Sakana AI's PC-ALM Trains 1,000-Layer Networks Without Backpropagation - AlphaSignal
- Augmented Lagrangian Predictive Coding: training 1000-layer networks without backpropagation - Sakana AI
- Sakana AI Researchers Introduce PC-ALM, a Layer-Local Alternative to Backpropagation That Trains 1000-Layer Networks - MarkTechPost
- Sakana AI | AI Companies - AlphaSignal
