NVIDIA's CUDA Moat Just Cracked. It Took Two Lines of Code.

There's a scene in every heist movie where someone discovers the safe was never locked. Not cracked — just... open. That's roughly what happened to NVIDIA's GPU monopoly this week.

A company called Wafer just served GLM-5.2 — a 744-billion-parameter frontier model — on AMD MI355X GPUs at 2,626 tokens per second per node. That's 80% of what a B200 does, at roughly half the cost. And the gap between "AMD can't run this" and "AMD runs this at near-parity" took, by their own account, two lines of code.

One was a quantization prefix mismatch. The other was a missing #ifdef USE_ROCM guard in a CUDA header. That's it. The CUDA moat — the thing NVIDIA has spent a decade building, the thing that supposedly made their GPUs unassailable — eroded by a single conditional compilation flag and a renamed module path.

Let's back up.

The Setup: Why AMD Was Always the Answer Nobody Wanted

Here's the dirty secret of AI infrastructure in 2026: there aren't enough NVIDIA GPUs. Frontier models drop every other week. Claude Fable, GLM-5.2, MiniMax M3 — each one demands more compute than the last. The token craze is getting crazier, and Blackwell supply can't keep up. So NVIDIA prices climb. And climb. And tokens get expensive.

AMD's Instinct MI350 series competes with Blackwell at the silicon level. On paper, the MI355X is a beast — 288GB of HBM3e, massive memory bandwidth, and at roughly 2.75x cheaper per GPU than a B300, the economics are almost absurd.

But nobody wanted to use it. And the reason was never the hardware.

It was software. NVIDIA's CUDA ecosystem is a decade-plus of accumulated advantage — kernels, libraries, day-zero support for every new model. When a new model drops, NVIDIA has an optimized path ready. AMD? You're lucky to find a container image that runs the thing at all. By the time you've spent weeks engineering a working deployment, the next model has already shipped, and AMD is playing catch-up again.

This is what the industry calls the "CUDA moat." It's real. It's been NVIDIA's single most powerful competitive advantage — more than the silicon itself. And for years, it worked exactly as designed.

What Wafer Actually Did

Wafer took GLM-5.2, quantized it to MXFP4 (a 4-bit floating point format), and served it on AMD MI355X capacity from TensorWave using the sglang inference engine.

The benchmark numbers, on a 20k input / 1k output workload with 60% cache hit rate:

So AMD hits about 80% of NVIDIA's throughput at less than half the cost. On performance-per-dollar, AMD wins decisively. This isn't a tie. It's a knockout in the only metric that matters to anyone paying the bill.

But the interesting part isn't the benchmark. It's what it took to get there.

Two Lines of Code

Wafer enabled speculative decoding on sglang to boost single-stream throughput. The sglang ROCm image doesn't support this out of the box. Two bugs stood in the way:

Bug 1: The MTP (multi-token prediction) head — the thing that makes speculative decoding work — keeps its shared expert in bf16, not the 4-bit MXFP4 format used for the rest of the model. But AMD's quantization tool (Quark) registered the shared expert under the main decoder's prefix (model.layers.78.mlp.shared_experts.*), while sglang looks for it under a different prefix (model.decoder.*). The mismatch means sglang's quantization lookup fails, defaults to building that expert as MXFP4, then crashes at load when it tries to stuff a full-width bf16 weight into a half-width 4-bit slot. Fix: copy the layer-78 entries to the un-quantized list a second time under the name sglang actually uses.

Bug 2: Deep speculative decoding (the 5/1/6 draft config that GLM recommends) was blocked because a fused multi-step metadata kernel needed for draft depth ≥4 had #include <cuda_runtime.h> with no ROCm guard. Fix: one #ifdef USE_ROCM.

Two trivial changes. Close to a 3x gain in single-stream throughput unlocked by a renamed module prefix and a preprocessor directive.

That's not a moat. That's a speed bump.

There was more engineering after that — tuning MoE kernel selection for GLM's specific fp4 shapes, switching from TP8 to TP4×DP2 for better prefill performance, enabling FP8 KV cache. Real work. But none of it required custom kernel development. None of it required reinventing the stack. It was configuration, tuning, and bug-fixing — the kind of work any competent inference team does in a sprint.

As Wafer themselves put it: "SOTA on AMD is becoming more a matter of support, not software. The CUDA moat is eroding in real time."

The Commenters Pushed Back. They Have a Point.

The Hacker News thread (160 upvotes, 46 comments) didn't let Wafer off easy. The most substantive pushback was on quantization quality.

The "lossless" MXFP4 claim — that 4-bit quantization doesn't degrade model quality — drew immediate skepticism. One commenter: "While cool, quantization to FP4 is practically never lossless in actual use. A lot of providers are advertising high TPS on Kimi and GLM, but the models are functionally lobotomized and no longer close to frontier quality." Another pointed out that Wafer's own numbers show accuracy degradation: GPQA-Diamond drops from 0.9217 to 0.9026 (−0.019), GSM8K drops from 0.965 to 0.955.

These are real degradations. "Lossless" is doing a lot of heavy lifting in that headline. On narrow benchmarks, the differences are small. In long-context reasoning, agentic workflows, and edge cases? Nobody's published the comparison. And that's where it matters.

The second pushback was on power. The MI355X pulls 1,400W per GPU versus the B200's 1,200W — about 16% more. But as one commenter correctly noted, raw wattage isn't the metric. Performance-per-watt is. And there, AMD is roughly 20-60% worse depending on the workload. So AMD wins on capital cost but loses on operating cost. The picture is more nuanced than "2x cheaper."

But here's the thing: none of these objections change the conclusion. The question was never "is AMD better than NVIDIA right now." It's "is AMD close enough that the cost difference makes it the rational choice for most workloads?" And that answer flipped this week.

Why This Actually Matters

Three reasons.

First: the supply problem is real and getting worse. NVIDIA can't make enough Blackwells. If you're building an AI product, you might wait months for allocation. AMD's MI355X is available now. Meta is deploying them. OpenAI signed a deal. Wafer alone has serviced over 700 customers on MI300x. The supply chain gap is closing from both directions — AMD making more chips, NVIDIA unable to make enough.

Second: the software gap is the only gap, and it's closing fast. Every bug Wafer fixed is now fixed for everyone. The fixes will be upstreamed. The next team that deploys GLM-5.2 on MI355X won't hit those bugs. The ROCm ecosystem is compounding — each deployment makes the next one easier. Meanwhile, AMD's upcoming MI400 series is already in testing.

Third: NVIDIA's response is a roadmap, not a solution. Commenters in the thread noted that Rubin — NVIDIA's next generation after Blackwell — will reportedly offer 22TB/s memory bandwidth versus Blackwell's 8TB/s. That's a massive jump. But Rubin isn't shipping yet, and when it does, it'll be expensive and allocation-constrained, exactly like every NVIDIA flagship before it. The performance lead will likely hold. The cost advantage will not.

The Bigger Picture: Agents Need Cheap Inference

Here's where this connects to everyone, not just data center operators.

AI agents — the kind that browse the web, run terminal commands, read your files, automate your workflow — make a lot of API calls. We're talking 30, 50, sometimes 100 tool calls per task. Each one is a round-trip to a model. At NVIDIA-priced tokens, that gets expensive fast. We've written about this before: the $300/month problem is real.

Cheap inference on AMD hardware changes the math. If tokens cost half as much, agents can do twice as much work for the same budget. Or the same work for half the cost. The entire economics of AI automation shift downward. The bottleneck moves from "can we afford to run this agent?" to "what should the agent do next?"

This is why the AMD story matters even if you never touch a GPU directly. The cost of intelligence is set by the cost of inference. The cost of inference is set by hardware supply. And the hardware supply story just got a second player who's close enough to matter.

The Honest Take

Wafer's post is partly marketing — they're an AMD-first inference provider, and they have a horse in this race. The "lossless" quantization claim deserves the skepticism it got. The power consumption gap is real and shouldn't be hand-waved.

But strip away the hype, and the core finding holds: a frontier model now runs on AMD hardware at near-parity performance and dramatically better cost, and the engineering effort to get there was measured in days, not months. That wouldn't have been true a year ago.

The CUDA moat isn't dry. But it's draining. And every two-line bugfix is another crack in the wall.

---

If you're building with AI agents, inference cost is your silent tax. CopperRiver runs open-source models — GLM, DeepSeek, Qwen, and more — directly on your Mac or through optimized cloud endpoints. No NVIDIA tax required. Check it out.

---

Related Reading

• Better Models, Worse Tools: Why Frontier AI Tool-Calling Is Quietly Regressing
• Meta Is Scared Its Own Engineers Will Leak Claude's Brain Into Llama
• The Model Is Not the Moat: Sakana Fugu and the End of AI Vendor Lock-In
• Microsoft's AI Tax: Copilot Price Hikes and the Forced-AI Business Model