Someone Ran a 2.78-Trillion-Parameter AI Model on a MacBook Pro

Someone ran a 2.78-trillion-parameter AI model on a MacBook Pro

There's a project on GitHub called WASTE. The name is an acronym — Weight-Aware Streaming Tensor Engine — but it's also a statement about what it represents: every token you send to a cloud API is paid for twice. Once on the invoice, and once in the electricity of a datacenter running a model that could, barely and awkwardly, run on hardware already sitting on your desk.

What WASTE does is take Kimi K3 — a 2.78-trillion-parameter open-weights model released by Moonshot AI in July — and run the full, unmodified thing on a consumer laptop. Not a distilled version. Not a pruned version. Not a "smaller model inspired by." The actual model, all 2.78 trillion parameters, answering questions on a 64 GB MacBook Pro.

The speed? About half a token per second. Twenty-six seconds for "The capital of Italy is Rome."

And that's not the point.

Why this matters

Let me explain why half a token per second is one of the most exciting numbers in AI right now.

Kimi K3 is a mixture-of-experts model. That means out of its 2.78 trillion parameters, only about 47 billion are active for any given token. The rest — roughly 96% of the model — is sitting idle at any given moment. Traditional inference loads the entire model into RAM because you don't know which experts you'll need until the moment of computation.

But here's the insight that makes WASTE work: idle parameters don't need to be in memory. They need to be reachable in time.

Think of it like a library. You don't need every book on your desk to write an essay. You need the few you're referencing right now, and you need to be able to find the next ones quickly. WASTE keeps the model's trunk — the shared architecture that every token passes through — resident in RAM. The experts, those 2.78 trillion parameters that are mostly sleeping, live on your SSD. When a token needs a specific expert, WASTE reads exactly that expert from disk in a single operation. One read. Not three. Not a seek per matrix.

The model is 982 GB on disk. It runs in 29 GB of RAM. On a 64 GB machine, it manages a 17 GB expert cache on top of the trunk.

The engineering is genuinely brilliant

This isn't just "put the big file on disk and read from it." The technical details in the WASTE repository reveal a level of systems engineering that's rare in the AI world, where most optimization work happens at the GPU cluster level.

The engine is written in C — 6,000 lines, zero dependencies. No BLAS, no CUDA, no Python at runtime. Just libc and pthreads. It's the kind of code that reads like it was written by someone who has opinions about memory alignment and isn't apologizing for it.

The core trick is something they call "router lookahead." Here's how it works. In a mixture-of-experts model, each layer has a router that decides which experts to activate. The router for layer L+1 normally can't run until layer L finishes — it needs the hidden state that layer L produces. So you can't preload the next layer's experts because you don't know which ones you'll need.

WASTE cheats. At the end of each layer, once its own disk reads are done, it runs the next layer's router using the current layer's hidden state — one residual early. It's a guess. But it's a good guess: 92% accurate at predicting the top expert, 81% accurate across the first six.

That guess lets the engine start fetching the next layer's experts before the current layer's computation finishes. The disk reads overlap with the math instead of blocking it. The result: the cache hit rate jumps from 14% to 38%, and the total bytes read don't change at all. Those records were going to be read anyway — the lookahead just moves them earlier.

This is the kind of optimization that separates people who understand computer architecture from people who throw GPUs at problems. The bottleneck isn't computation. It's I/O. And I/O can be hidden with pipelining, which is a technique older than most AI researchers.

The quantization story

The other piece of the puzzle is compression. The published K3 weights are 1.42 TB. WASTE converts them to 982 GB using residual vector quantization: three stages of 256-entry codebooks over 8-dimensional vectors, at 3 bits per weight.

Three bits. That's how much information each parameter takes up. And the model still works because Kimi K3 was trained with quantization-aware training on the experts specifically. They built tolerance for compression into the model during training, not after.

The trunk — the shared architecture — stays at 4 to 8 bits. They tried a 3-bit trunk, measured it, and the output collapsed. The trunk wasn't trained for that level of compression, so it doesn't survive it. Honest engineering: they measured, it failed, they documented why, and they moved on.

What this means for local AI

Here's why you should care about a project that generates text at 0.5 tokens per second.

WASTE proves something that was theoretical until now: frontier-scale open models can run on consumer hardware. Not fast. Not efficiently. But they run. The constraint is no longer "can it be done?" but "how fast can we make it?"

And speed is an engineering problem. Engineering problems get solved.

The WASTE README is remarkably honest about where the performance levers are. The two optimizations that looked biggest — reading fewer bytes per token and keeping more in RAM — both turned out to be dead ends. What actually worked was timing: overlapping reads with computation, and guessing ahead. Those are software techniques that improve with better implementations, better schedulers, better hardware.

The SSD in a MacBook Pro reads at 12.78 GB/s. Next year's will be faster. The NVMe spec isn't standing still. And while half a token per second isn't usable for real-time chat, it's absolutely usable for batch processing, background tasks, and workflows where you ask a question and check back in a minute.

More importantly, this isn't just about Kimi K3. The engine runs Kimi-Linear-48B from a 19 GB container at 10.7 tokens per second — that's a genuinely usable speed for a capable model. The streaming architecture scales down naturally. K3 is simply the hardest case that exists today.

The deeper shift

For the last two years, the implicit assumption in AI has been that frontier models require frontier infrastructure. Massive GPU clusters. Data center power contracts. Networking fabrics that cost more than most companies' annual revenue. If you want a capable model, you pay per token to someone who owns that infrastructure.

Open-source models have been chipping away at this assumption. Qwen, DeepSeek, GLM — these are real models with real capabilities, available as downloadable weights. But running them locally has meant either using smaller models or accepting significant quality reductions through aggressive quantization.

WASTE points at a third path. Not smaller models. Not compressed models. Full-scale models, streamed intelligently from storage you already own.

The economics are striking. Kimi K3 on a cloud API costs money per token. Kimi K3 on your laptop costs electricity. The WASTE README puts it plainly: the name means "to be the first concrete step toward ending that waste of tokens."

There's a philosophical dimension here that resonates. Cloud AI is convenient, but it comes with strings. Your data leaves your machine. Your usage is tracked. Your access depends on a company's business model, pricing decisions, and uptime. When OpenAI had models escape their sandbox in July, it was a reminder that the infrastructure layer of AI is fragile in ways we're still discovering.

Running a model locally — even slowly — means none of those strings are attached. No network. No per-token invoice. Nothing leaving the machine. The difference between "you may not send that data to an API" and "run it here."

Where this goes next

WASTE is version 0.x. A proof of concept written for one model, on one architecture, by what appears to be a small team. The Metal backend exists but is 22% slower than CPU because the workload — hundreds of small dependent matrix operations per token — is the worst possible shape for a GPU. The AVX-512 path compiles but hasn't been tested because no CI runner has the hardware.

These are gaps, not walls. The fundamental insight — that mixture-of-experts models can stream from NVMe because most parameters are idle — is architecture-independent. It works because of a property of the model (sparse activation), not a property of the hardware (any specific chip).

Expect to see this approach refined. Faster SSDs, better caching strategies, smarter prefetching, and purpose-built inference engines will push the speeds up. The 0.5 tok/s of today could be 5 tok/s within a year, and 5 tok/s is the threshold where local frontier models become genuinely practical for most use cases.

The implication is profound. If the trend in open-source AI continues — capable models, open weights, and now proof that they can run on hardware you already own — the cloud API business model faces real pressure. Not because cloud APIs are bad, but because the alternative is getting better faster than most people realize.

The bottom line

A 2.78-trillion-parameter model running on a laptop is a line in the sand. It says the future of AI isn't necessarily a subscription to someone else's data center. It can be a model file on your machine, running at whatever speed your hardware allows, answering your questions without telling anyone what you asked.

That future is slower than cloud AI. It's clumsier. It requires technical knowledge that most people don't have yet. But it exists, it works, and the gap between "barely runs" and "runs well" is the kind of gap that closes.

The next time someone tells you frontier AI requires a billion-dollar data center, remember WASTE. Remember that 96% of a trillion-parameter model is asleep at any given moment, and that idle weight doesn't need to be in memory. It just needs to be reachable.

---

If the idea of running powerful AI models on your own hardware excites you, CopperRiver is built around exactly this philosophy. It's a desktop AI assistant for Mac that uses open-source models to browse the web, run terminal commands, read files, and automate tasks — all locally, all private, all yours. Plans start at $9/month.