You're Spending 95% of Your RL Training Budget on Layers That Don't Matter

Here's something that should make you uncomfortable.

When you run reinforcement learning post-training on a large language model — the step that turns a base model into something that can actually reason, code, and follow instructions — you're updating every single layer. All 28 of them. Or 36. Or 80. Billions of parameters, all getting gradient updates, all consuming GPU time, all fighting for memory bandwidth.

A new paper from researchers at the University of Minnesota and Amazon, published July 2nd on arXiv, shows that you could get the same results by training just one layer.

Not "almost the same." Not "comparable." In some cases, training a single layer outperformed full-parameter training. As in: better accuracy. Fewer parameters touched. Less compute wasted.

Let that sink in for a second.

The Experiment

The setup is beautifully simple, which is part of why this paper hit the front page of Hacker News within hours of going live.

Take a pretrained model — they used Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen2.5-Math-1.5B, Qwen2.5-Instruct variants, and DeepSeek-Distilled-Qwen-7B. Then do RL post-training using GRPO (or one of its variants). Standard stuff. This is the kind of training that teaches models to reason through math problems, write working code, and make decisions in agentic environments.

But here's the twist: instead of updating all layers simultaneously, freeze everything except one. Train just that single layer with the same RL algorithm, same dataset, same hyperparameters. Then unfreeze a different layer, freeze everything else, and repeat. Layer by layer, across the entire model.

The question they were asking: how much of the total RL improvement does each individual layer capture when trained alone?

They called this metric "layer contribution." It's simply the fraction of full-parameter RL gains that a single layer can recover on its own. A score of 1.0 means the layer alone captures 100% of what full-parameter training achieved. Above 1.0 means it actually did better.

What They Found

The results are consistent enough to be unsettling.

One layer recovered up to 114% of the gains from full-parameter RL training. That's not a typo. Training one layer — out of 28 — sometimes beat training all of them combined.

The best layers — consistently, across every single model they tested — sat in the middle of the transformer stack. Around 40-60% of the way through the network. Layers near the input and output? Barely contributed at all. The weakest layers recovered less than 30% of the gains.

On Qwen3-1.7B specifically, layer contribution ranged from 0.28 (Layer 24) to 1.14 (Layer 10). Five out of 28 layers exceeded 1.0. Seven layers fell below 0.5. That means roughly a quarter of the model's layers are essentially dead weight during RL, and another quarter are actively capable of doing the whole job themselves.

And this wasn't a fluke with one model or one training setup. The pattern held across:

• Two model families (Qwen3 and Qwen2.5)
• Three different RL algorithms (GRPO, GiGPO, and Dr. GRPO)
• Multiple task domains (mathematical reasoning, code generation, and agentic decision-making via ALFWorld)
• Seven different model configurations total

The layer rankings — which layers matter most — stayed remarkably stable. They correlated strongly across different datasets (Spearman correlation above 0.9 in some cases), different tasks, and different model scales. If Layer 10 was the hero in Qwen3-1.7B trained on NuminaMath-CoT, the corresponding middle layers were the heroes everywhere else too. The model "knew" which layers to use for learning, and it was always the same neighborhood.

Why This Matters

There are two ways to read this, and both are important.

The scientific reading: We've been assuming that RL post-training works by making coordinated adjustments throughout the entire network. Every layer learns a little bit, the model gets smarter, everyone's happy. That assumption appears to be wrong. RL gains are concentrated in a small, predictable subset of layers. The rest of the network is essentially along for the ride — and in some cases, it's actively getting in the way. When you train all layers together, the low-contribution layers can dilute the improvements that the high-contribution layers would achieve on their own.

That's a big deal for our understanding of how these models actually work. It suggests that the "intelligence boost" from RL isn't a network-wide phenomenon. It's localized. The middle layers are where the actual learning happens. The outer layers are frozen knowledge.

The practical reading: This is free performance sitting on the table.

If you're fine-tuning open-source models — and a lot of people are right now — you're spending GPU time updating parameters that don't need updating. The researchers showed that simply prioritizing the high-contribution layers consistently outperformed standard full-parameter training. Same model. Same data. Better results.

The "Just Train the Middle" Heuristic

Here's the part that should make open-source model developers drop what they're doing.

You don't even need to run the expensive layer-by-layer profiling to benefit. The pattern is consistent enough that a dumb heuristic works: just train the middle layers.

The researchers tested exactly this. Instead of scanning all layers to find which ones matter, they simply selected layers from the middle portion of the network. No profiling. No measurement. No analysis. Just "train layers 40%-60% of the way through the stack." And it worked. Comparable or better performance than full-parameter training, with zero additional overhead.

On Qwen3-8B, training only the ten highest-contribution layers achieved 69.1% average accuracy on mathematical reasoning benchmarks. Full-parameter training on the same model? 66.4%. That's better results with roughly a quarter of the parameters trained. Less compute. Less memory for optimizer states. Faster wall-clock training time.

That's the kind of finding that goes from a research paper to a default training configuration within months. It's too simple and too effective to stay in academia.

Ensembles of Specialists

There's one more finding that opens an entirely new direction, and honestly it might be the most interesting part of the whole paper.

When the researchers trained different individual layers, the resulting models weren't just slightly different versions of the same thing. They had complementary strengths. Layer 10's model solved different subsets of problems than Layer 14's model. They made different mistakes. They excelled at different things. Same base model, same training data, same RL algorithm — but different layers produced models with genuinely different capabilities.

Combine them through simple majority voting, and you get additional gains beyond what any single model — including the full-parameter baseline — could achieve alone.

This suggests something profound about transformer architecture: the layers aren't a monolithic learning system. They're more like a team of specialists, each capable of independent improvement, each with different blind spots. RL post-training as we currently do it is like sending the entire team to the same training seminar when really, each member just needs to work on their own area.

What This Means for Open-Source AI

The timing of this paper matters. The open-source AI community is in the middle of an RL post-training boom. Everyone and their startup is running GRPO on Qwen, DeepSeek, and Llama variants. Compute costs are the bottleneck. GPU hours are the budget line item that kills projects.

If you can train fewer parameters and get better results, that's not an incremental improvement — it's a paradigm shift.

For smaller teams and independent developers, this could be the difference between "we can afford to post-train one model, once" and "we can afford to iterate." Training 10 layers instead of 36 isn't just a 3.6x speedup on paper. It means smaller gradient states, less optimizer memory, and the ability to use larger batch sizes on the same hardware. The practical speedup compounds in ways that the raw parameter count doesn't capture.

For the big labs, it raises an uncomfortable question: how much of the billions spent on RL training has been wasted on updating parameters that don't matter? If your RL training runs cost tens of millions of dollars per cycle, the idea that most of your layers are dead weight during RL is not a fun conversation to have with your CFO.

The Skeptical Take

It's worth being honest about what this paper doesn't claim.

The experiments focus on RL post-training specifically — not pre-training, not supervised fine-tuning. The findings might not generalize to every training paradigm. Supervised fine-tuning involves different optimization dynamics, and the layer concentration effect could look different there.

The models tested are all in the 1.5B to 8B parameter range. That's substantial but not frontier-scale. Whether a 400B parameter model exhibits the same layer concentration during its RL phases is unknown. The physics might change at scale — though the consistency across 1.7B, 4B, and 8B suggests the pattern is structural, not scale-dependent.

Also, the "layer contribution" metric measures how well a layer performs in isolation. In full-parameter training, layers interact. They adapt together, compensate for each other. The fact that one layer can do the job alone doesn't necessarily mean the others are useless in a joint setting. They might be doing subtle work that doesn't show up in benchmark metrics.

But even with those caveats, the core finding is robust across seven models, three algorithms, and multiple task domains. The consistency is the story. This isn't one lucky experiment on one lucky model. It's a structural property of how RL reshapes language models. And it's been hiding in plain sight because nobody thought to check.

The Bottom Line

We've been treating transformer layers as a uniform substrate during RL post-training. Update them all, let the optimizer sort it out, hope for the best. This paper suggests that's fundamentally wrong. RL post-training has a preferred neighborhood, and it's always the middle of the network.

If you're building with open-source models, this is worth tracking closely. The transition from "train everything" to "train the layers that matter" could make fine-tuning dramatically cheaper and more effective. And for tools that run these models locally — where every bit of efficiency counts — that's not academic. It's the difference between a model that fits on your machine and one that doesn't.

The paper is open access on arXiv (2607.01232). If you're hands-on with model training, the layer-adaptive strategies in Section 5 are the practical takeaway. Start experimenting with training subsets of layers. The results might surprise you — they surprised everyone else.

---

CopperRiver runs open-source AI models — GLM, DeepSeek, Qwen, and more — right on your Mac. No cloud, no API keys, no per-token costs. Check it out.

---

Related Reading

• Compounding Correctness: Why Spending More Tokens Now Actually Makes AI Better
• A 9B Model Just Beat 35B Models at Coding. The Self-Improvement Method Behind It Is the Real Story.
• DeepSeek's DSpark Just Made AI 85% Faster — With Software, Not Silicon
• Qwen AgentWorld: The AI Model That Simulates Reality (And Beats GPT-5.4 Doing It)