The Best Backprop Killer Yet Explains Why Backprop Won't Die

Every few months, someone declares backpropagation obsolete. The ritual is predictable: a paper drops, the headline says "trains neural networks without backprop," HN argues for a day, and every training run on Earth still uses backprop by Friday. This week's entrant is PC-ALM from Sakana AI — Augmented Lagrangian Predictive Coding, by Jeffrey Seely and Julian Gould — and it trains residual MLPs up to 1,000 layers deep, nearly matching backprop, using only layer-local computation. It hit HN's front page Monday and it's the real deal.

It's also, by its own admission, not a backprop replacement. And that's exactly why it's worth your attention — because what it actually attacks is the constraint quietly strangling AI right now: backprop's memory bill.

The tax backprop has been charging you

Backpropagation has a hidden cost that has nothing to do with gradients or math. It's a timing and storage tax. To train with backprop, a network must run in three strictly ordered phases: forward pass, backward pass, weight update. Every layer has to store its activations and then sit there, frozen, waiting for the error signal to propagate all the way back. Layer one cannot update until layer one thousand has been visited.

Sakana's post calls this the "phase locking" problem, and it's one of the standard arguments for why the brain can't be doing literal backprop — there's no known biological mechanism to enforce that kind of strict, network-wide timing coordination. A neuron in your visual cortex can't hold its activation hostage while an error signal treks back from your motor cortex.

The practical version of this tax is memory. Stored activations scale with depth and with sequence length. It's why training long-context models is a memory nightmare, why gradient checkpointing exists (recompute activations rather than store them — literally trading compute for memory), and why the KV cache is the thing that kills your inference bill. Backprop's global freeze means global state.

What PC-ALM actually does

Predictive coding (PC) is the long-running research program trying to fix this: let each layer talk only to its immediate neighbors, and let learning emerge from local prediction errors instead of a global backward pass. Beautiful idea, documented flaw — the credit signal decays as it diffuses through layers. In deep, narrow networks, the signal that tells layer 3 "you contributed to the error" gets faint before it arrives. PC works fine on wide shallow nets and falls apart as you go deeper.

PC-ALM's fix is a control theorist's fix, which is what makes it interesting. Each layer gets a set of dual neurons — Lagrange multipliers, one per activation — that accumulate local prediction error over time. In dynamics terms, every layer becomes a PI feedback controller: the current prediction error is the proportional term, the accumulated dual variable is the integral term. Global credit assignment emerges from a mesh of purely local controllers.

The kicker is the theory: in linear networks, these dual variables converge to the exact credit signals backprop computes. Not an approximation — the same gradients, recovered by iteration instead of a global backward pass. The paper traces this connection to a 1988 result by LeCun, and derives it fully. On their propagation plots, PC's credit signal creeps inward like heat diffusion; PC-ALM's drives through the network as a wavefront, which they charmingly call "ballistic" propagation.

And the parity isn't a one-off. Across the full width-by-depth grid on Fashion-MNIST — every activation function they tested — PC-ALM tracks backprop while standard PC degrades exactly where you'd expect, as networks get deeper and narrower. The gains are largest precisely where PC fails hardest, at no extra inference cost. The dual update is doing all of the work.

The headline result: a 1,000-layer residual MLP on MNIST, trained entirely with local dynamics, landing within roughly two percentage points of backprop across the whole depth range. PC collapses long before that. No extra inference budget — matched compute, matched seeds, only the dual update differs.

Now read the fine print

Here's where the contrarian part of this essay lives, because the "backprop alternative" framing fails in both directions.

First, it fails as a challenge. The benchmarks are MNIST, Fashion-MNIST, CIFAR-10, and Tiny ImageNet — MLPs and a ResNet-18. That is 2015-era territory, and the authors are refreshingly upfront about it: they follow the predictive-coding literature's standard tasks because that's the regime where PC methods can be compared at all. Early HN readers digging through the benchmark figures clocked the absolute CIFAR-10 accuracy around the mid-70s — respectable, not competitive. On ResNet-18, PC-ALM improves over standard PC on every task and narrows the gap to backprop; "narrowing the gap" is doing honest work in that sentence, because closing it is not what happened. The 1,000-layer parity claim is parity at width 32, five epochs — a configuration chosen to be stable at depth, not to top leaderboards. And each training step requires T = 2L serial inference iterations, one wavefront sweep per update. On a GPU, that's just slower than backprop's one forward, one backward. The paper doesn't hide this. The authors state plainly that replacing backprop isn't their goal — their motive is understanding how distributed systems like brains can compute gradients without global coordination.

Second — and this is the mistake in the other direction — dismissing it as scaled-down science fair work misses what changed. Every previous "local learning" result decayed with depth in principle. PC-ALM is the first method to show the decay isn't a law of nature, that exact credit signals can be recovered from local dynamics with no memory of the network's global state. The question "does it beat backprop on CIFAR?" was always the wrong question. The right one: what does training look like when no layer ever needs to store activations for a global backward pass?

The memory bill is the actual story

Strip away the neuroscience framing and PC-ALM is an existence proof for gradient descent with an O(neighbors) memory footprint. That's the prize. Not speed — backprop on a GPU is nearly unbeatable at what it does. But backprop's cost structure assumes you can afford global synchronization: one giant accelerator cluster, one phase-locked training run, every activation stored until the backward pass releases it.

Industry has already been paying down this tax with workarounds. Gradient checkpointing is the tell: we voluntarily recompute activations because storing them is more expensive than calculating them twice. That's a bet that compute stays cheap and memory stays dear — a bet that has kept winning because NVIDIA keeps making it win. Local learning is the same bet taken to its logical extreme: never store anything globally at all.

The places that bet pays off first aren't frontier pretraining:

• Continual, on-device learning. A phase-locked system can't learn while it runs. A locally-coupled one can — there's no global freeze to violate. One of the sharpest observations in the HN thread was simply: "it lets you update without pausing the entire system."
• Fine-tuning under tight memory. The same commenter spectrum — LoRA at one end, full fine-tuning at the other — has an unfilled slot for "fine-tune big models without materializing the backward pass." No evidence PC-ALM works on transformers yet. But it's now a research direction with a theoretical spine, not a wish.
• Neuromorphic hardware, which Sakana names explicitly. Chips that compute by settling into dynamics rather than executing scheduled passes can't run backprop at all. They've been waiting forty years for exactly this kind of algorithm.

What to do Monday morning

Nothing, if you run a training stack — your backward passes are safe for years. Three things if you're paying attention:

One, read the paper for the linear-network proof and the control-theory framing; the derivation that dual variables converge to backprop's credit signals is the load-bearing wall for everything that follows in this research line.

Two, note the velocity. The code is on GitHub, MIT-licensed, Python, already past a hundred stars within a day of the HN thread. If a transformer-scale result exists in this direction, someone will attempt it publicly within months, and now you know what to watch for: not "beats backprop" but "trains a real model with bounded activation memory."

Three, recalibrate the backprop-killer ritual itself. The reason this entrant feels different isn't the 1,000 layers — it's the honesty. A paper whose benchmarks say "we are small on purpose" and whose theory says "the signals are exact in the linear case" is how actual paradigm shifts start. The loud ones claiming to dethrone backprop were selling hype. This one is selling a constraint relaxation: global synchronization, it turns out, was never a mathematical requirement for gradient computation. It was a hardware convenience.

The brain never ran backprop, and for forty years that looked like evolution's compromise — biology making do without the good algorithm. PC-ALM quietly suggests the opposite reading: local wiring plus iteration was always capable of exact gradients, and the global backward pass is just what you build when transistors are cheaper than time. Backprop's memory tax is now officially optional in principle. Collecting on that principle will take years and probably new silicon. But the invoice has been disputed.