Your Local LLM Isn’t Dumb. Your Inference Stack Is.

You know the ritual. Someone on the internet swears a model is incredible — "frontier-level, just try it." You download it. It writes three lines of nonsense, forgets your instructions halfway through, and mangles a tool call. You close the tab and file it away as proof that open models aren't ready.

Here's the uncomfortable part: you probably weren't reviewing the model. You were reviewing your inference stack. And your inference stack is a snowflake.

Last week, a user named thr3e posted a two-part experiment series on the Level1Techs forum called "Why your local LLM feels dumber than it is." It hit the Hacker News front page on Friday, picked up 214 points, and the comment section filled with people quietly realizing they'd been blaming the wrong suspect for years.

The post asks a question nobody really wants the answer to: when you run the exact same weights as the lab that trained them, how different is the output, actually? Not "does it feel different." How many tokens come out different, measured, at the logit level?

The answer is going to ruin your afternoon.

The setup, because the rigor matters

This wasn't someone pasting prompts into Ollama and vibes-matching the replies. The author took the official BF16 checkpoint of Qwen3.6-27B, ran it on an RTX PRO 6000 Blackwell with a pinned nightly vLLM build, tensor parallelism 1, eager execution, CUDA graphs and prefix caching and speculative decoding all disabled. One GPU, one model, everything held still.

The workload is the part I love most. Not needle-in-a-haystack synthetics. Not MMLU questions the model has seen a thousand times. It's a roughly 100k-token context dump from a real workstream — actual tool calls, actual work products — deliberately chosen because it doesn't exist in any benchmark or training scrape. Nobody could have gamed it. Nobody could have calibrated a quant around it.

Then, during prefill, they captured the full-vocabulary logits every 32 tokens and computed the divergence offline in FP64. The headline metric is "top-1 flips": positions where, given the identical token history, a configuration would have picked a different next token than the baseline. Greedy argmax, head to head.

That's about as clean as this kind of experiment gets. And the results are brutal.

Finding one: swap the attention kernel, get a different model

vLLM lets you pick your full-attention backend: FlashAttention 2, FlashInfer, Triton. Same weights, same GPU, same prompt. The only change is which CUDA kernel does the math during prefill.

First few thousand tokens: total agreement. Then, deeper into the prompt, the backends start disagreeing about what the next token should be. Not smoothly, not proportional to context length — in clusters, tied to what the prompt was actually about at that position.

Before anyone cries noise: the control runs are the best part. Same backend, run multiple times, every hidden state — bit for bit identical. The divergence comes exclusively from the matrix multiplications and additions inside the kernels themselves. Different attention backend, different arithmetic order, different rounding, different tokens out.

Sit with that. The model is the same. The math library doing the multiply-adds is different. The model's behavior changes. There is no "the model" independent of the stack executing it — there's only model-plus-kernel, and every stack on earth is a slightly different one.

Oh, and the vLLM nightly container the author used? 734 Python packages. Each with its own bugs and undocumented quirks. Your inference "engine" is a small city of code, and your specific path through it is yours alone.

Finding two: the KV cache is where your IQ goes to die quietly

Next experiment: leave the weights in full BF16 glory and quantize only the KV cache — the running memory of everything the model has read so far. This is the knob people crank to fit longer contexts into less VRAM. It feels free. It is not free.

On a real tool-calling workload, BF16 sailed through. The int8 KV cache hit a tool-calling error — and eventually recovered. The int4 KV cache hit the same error and never recovered. Completely reproducible. The model with surgically intact weights was functionally broken because its scratch memory was compressed too hard.

This is the mechanism behind a thing everyone has felt and nobody could name: the model that's sharp for the first 30k tokens of your session and then slowly turns into a goldfish. It's not "context rot" mysticism. Sometimes it's literally your KV cache losing precision with every layer, every token, until the tool calls stop closing.

The forum post's own subtitle for this section is "why your LLM's IQ drops like a rock after 40k tokens," and I have bad news about how many "long context" setups out there are quietly running compressed KV.

Finding three: the quant bakeoff, or, trust no one's model card

Then the main event: five versions of the same Qwen3.6-27B, KV cache pinned at BF16, only the weights and activations differing.

• The official BF16 checkpoint (reference)
• Qwen's own official FP8 release
• A community INT8 (W8A16) quant by a Hugging Face user going by TheHouseOfTheDude — made in one shot, with no calibration dataset at all
• NVIDIA's official NVFP4 release
• A community AWQ 4-bit, calibrated on "STEM and Agentic" data

The results flip the expected hierarchy upside down. The community INT8 — the one with zero calibration effort — beats Qwen's own first-party FP8. NVIDIA's official NVFP4 release comes in dead last, hitting roughly 50% top-1 token flips by 88k context. Half the time, given the same history, it would have picked a different next token than the model it's supposedly representing.

It gets better. NVIDIA's "FP4" wasn't even running native FP4 arithmetic — vLLM detected the GPU lacked that path and silently fell back to weight-only compression through Marlin kernels. The label on the box and the computation inside the box were two different things, which is a pretty good summary of the quant ecosystem generally.

And the failures aren't cosmetic. Both 4-bit variants botched the actual work: they failed to close their tool calls correctly and ran the wrong Cisco command — show run instead of show arp. The FP8 and INT8 versions completed the same calls fine. Same model. Same prompt. Same hardware. The 4-bits executed the wrong command on a router.

The post also takes aim at the KL-divergence numbers quant authors love to flaunt on model cards — those "near-zero KLD!" badges. A KLD number is uninterpretable unless the author discloses the reference checkpoint, runtime, calibration data, context lengths, aggregation method, and direction of measurement. Most don't. The number is marketing with decimals.

So whose fault is your dumb local model?

Worth saying clearly: none of this means quants are a scam, or that 4-bit is always wrong. Plenty of people run Qwen 4-bit locally and get genuinely great results — one person in the HN thread reported a 4-bit build indistinguishable from a frontier-flash-tier model in their internal tests. The problem is that quality is workload-dependent in ways nobody can predict from the model card, and the failure modes cluster exactly where local AI is supposed to shine: long agentic sessions with real tool calls. Short chat? Fine. 90k tokens of actual work? That's where the flips pile up.

The HN thread added its own greatest hits of silent stack failure. The most useful confession: a huge fraction of "my local model is dumb" cases aren't the quant at all — they're the chat template. Plenty of GGUF conversions drop the template from the metadata, the runtime silently falls back to a generic one, the model still talks fluently so nobody notices, and it's just… noticeably dumber. The model runs fine and gets reviewed badly because a metadata field went missing.

Then there's the default-quant trap: the easy local runners hand you a Q4 by default, so plenty of people have genuinely never run the model they think they're running. And the sampler gotcha the forum author flags in passing — crank temperature too low and your Qwen sits there looping inside its own THINK block, which gets reported as "the model is broken" when it's a settings file.

By the time you've stacked the quant, the KV cache compression, the template fallback, the sampler defaults, and whatever kernel path your particular GPU generation happens to trigger, "this model is dumb" is a claim about a system you've never actually characterized.

A field guide to not fooling yourself

Boiling the post and thread down into rules you can actually use:

• Match the model card's sampler settings. Temp, top-p, chat template — they vary by model, and the card is the source of truth. If your Qwen loops in its thinking tags, your temperature is probably too low.
• Don't quantize the KV cache below int8 for anything agentic or long-context. If VRAM forces you to, know that you're trading away exactly the capability you're probably running local models for.
• Prefer Q8-class quants when you can afford them. Slower, bigger, boring, correct. The community W8A16 in this test was the most faithful quant in the entire bakeoff — better than first-party releases.
• Treat 4-bit as a budget option, not a default. Great when it works, and "when" depends on your workload in ways you can't read off the card.
• Grep your GGUF for the chat template before you blame the model. Silent fallbacks are everywhere.
• Benchmark your own workload. Not three zero-shot prompts at temperature zero. Replays of your real sessions, your real tool calls, your real context lengths. The author's point about labs benchmarking on reference hardware cuts both ways: their numbers describe their stack, and yours is a different organism.

The unifying idea, if you want one: an LLM is not a file. It's a file plus a stack, and the stack is making decisions — kernel selection, quant fallbacks, template defaults — that never appear in any changelog but show up in every output.

There's also a quieter, more optimistic read sitting in the HN thread. Someone put it better than I can: at least with local models, when quality drops, you can do something about it — pin a better quant, fix a setting, verify against reference. When a hosted model quietly gets worse, your only instrument is vibes, and your only recourse is a support ticket.

That control is the whole reason local and open models matter, and it's the reason CopperRiver runs on open-weights models in the first place — served at proper precision, pinned versions, no silent downgrades. If you want the open model without the stack roulette, that's literally the product. If you'd rather run the snowflake yourself, now you at least know which snowflakes bite.