What Is AI Model Quantization? (And Why It's the Key to Running AI Locally)

A 70-billion-parameter AI model needs about 140 GB of memory at full precision. That's two NVIDIA A100 GPUs — roughly $60,000 of hardware. Nobody's running that in their living room.

But here's the thing: the same model, quantized to 4-bit, needs about 40 GB. That fits on a single consumer GPU with some CPU offloading, or a beefy gaming PC with 64 GB of RAM.

That's not a minor optimization. That's the difference between "AI lives in a data center" and "AI lives on your desk." And the technique that bridges that gap — quantization — might be the single most important technology in the open-source AI movement. Without it, none of this works.

What quantization actually does

Computers store numbers. AI models are, at their core, gigantic collections of numbers — billions of them, called parameters or weights. By default, these numbers are stored in FP16 (16-bit floating point). Each weight takes 16 bits of memory.

Quantization is the art of storing those same numbers in fewer bits.

Going from 16-bit to 4-bit means each weight uses four times less memory. A 70B model that ate 140 GB now eats 35 GB. Same model, same architecture, same number of parameters — just represented more efficiently.

The obvious question: does that actually work? Can you cram the same information into a quarter of the space without the model getting dumber?

The answer is: mostly yes, with some important asterisks. And understanding those asterisks is the difference between a model that works great locally and one that gives you subtly wrong answers.

The reason it works: AI is memory-bound, not compute-bound

Here's something that surprises people: during inference (actually generating text), AI models are rarely limited by how fast they can do math. They're limited by how fast they can load numbers from memory.

Every single token the model generates, it has to read every single weight from memory. For a 70B model at FP16, that's 140 GB of data movement per token. The GPU's compute units are sitting there waiting for the memory bus to deliver the numbers.

Quantization helps twice: the weights are smaller (less memory used), and they're faster to load (less data movement per token). You get a memory win and a speed win from the same technique.

This is why quantization isn't a hack or a workaround. It's addressing the actual bottleneck in AI inference. If you can't move the data fast enough, making the data smaller is the most direct solution.

The precision ladder: from FP16 to 1-bit

Let's walk through the levels, because each one has a different personality.

FP16 / BF16 (16-bit) — Full precision

This is the baseline. Every weight is stored as a 16-bit floating-point number. Maximum quality, maximum memory. What the model was trained at (mostly — some are now trained in FP8 or BF16 mixed precision). This is what you get from OpenAI's API, Anthropic's API, any cloud-hosted model. You're paying for the full thing.

FP8 (8-bit float) — The enterprise sweet spot

Eight-bit floating point. Not integer — floating point, which means it can represent a wider dynamic range than INT8. FP8 is where the frontier labs are actually living. DeepSeek-V3 was trained natively in FP8. DeepSeek-R1 ships FP8 weights. On NVIDIA H100 and newer GPUs, FP8 matrix operations use specialized hardware (DeepGEMM kernels) that run 3–6x faster than general-purpose paths.

Quality loss at FP8? Essentially zero. We're talking under 1% on standard benchmarks. For production inference, FP8 is the new standard — you get 2x memory savings with no meaningful quality cost. The only catch: you need Hopper or Blackwell GPUs (H100, H200, B200). Consumer GPUs don't support it in hardware.

INT8 (8-bit integer) — The safe compression

Classic post-training quantization. Every weight gets mapped from a 16-bit float to an 8-bit integer using a scale factor. The scale tells you "each integer step represents X units in the original float space." At inference, you multiply back.

INT8 gives you 2x memory savings with negligible quality loss — typically under 1% on benchmarks. It works on older hardware (GTX 10-series and newer). The catch is that a small number of weights are "outliers" — values far from the mean that carry disproportionate importance. Naive INT8 squashes these outliers and causes disproportionate damage. Smarter methods like bitsandbytes' LLM.int8() detect and preserve these outliers at higher precision, getting the best of both worlds.

INT4 (4-bit) — Where things get interesting

Now we're talking about 4x compression. A 70B model drops from 140 GB to about 40 GB. This is the level that makes local inference of large models actually practical on consumer hardware.

But 4-bit is where you start paying attention to how you quantize. There are several competing methods, and they're not all equal:

GPTQ is the old reliable. It quantizes weights row by row, using a calibration dataset to minimize the error introduced by compression. The calibration process takes about 20 minutes on an A100 for an 8B model. GPTQ is battle-tested, widely supported, and good. The actively maintained successor is called GPTQModel (the original AutoGPTQ is deprecated).

AWQ (Activation-aware Weight Quantization) is the newer contender. Instead of treating all weights equally, AWQ uses calibration data to identify which weight channels matter most — the ones with high activation magnitudes — and preserves them at higher precision. It's faster to calibrate (about 10 minutes for 8B), and often edges out GPTQ slightly on quality benchmarks.

GGUF Q4_K_M is the llama.cpp variant — and it's what most people running local models actually use. The "K" in K-quant refers to a clever mixed-precision scheme where different parts of each weight block use different bit depths. Q4_K_M averages about 4.5 bits per weight but puts more precision where it matters. It's the community default for local deployment.

Quality at INT4? Good, with caveats. You lose about 1–3% on standard benchmarks like MMLU. For chat, summarization, creative writing, and general Q&A, you won't notice. For precise code generation or multi-step math reasoning, you might.

Sub-4-bit — The frontier

Below 4 bits, things get rough. Q3_K_M (about 3.4 bits per weight) shows noticeable degradation — 3–8% accuracy loss. Q2_K (2.6 bits) is usable but meaningfully worse. At 1.58 bits — the BitNet approach where every weight is just {-1, 0, 1} — you need to train the model from scratch in that format. It works, but it's a research project, not a deployment strategy.

Sub-4-bit quantization is where you end up when you absolutely must fit a model into a specific memory budget and quality is a secondary concern. It exists. It's useful for edge cases. It's not where you want to live.

GGUF: the format that made local AI possible

If quantization is the engine, GGUF is the chassis. It's a single-file format — you download one .gguf file and you have everything: the model weights, the tokenizer, the chat template, the metadata. No multi-file setups, no configuration hell.

GGUF was created by Georgi Gerganov for llama.cpp, and it's become the de facto standard for local AI. Ollama uses it. LM Studio uses it. Every major local LLM tool is built on it under the hood. The genius of GGUF isn't just the format — it's that llama.cpp can run on CPU, GPU, Apple Silicon Metal, or any combination of the three.

That hybrid capability is the killer feature. You load as many model layers as fit in your GPU's VRAM, and the rest spills over to system RAM and runs on the CPU. It's slower than pure GPU, but it means any model can run on any hardware. You're not gated by VRAM anymore — you're gated by total system memory.

This is how someone ran a 744-billion-parameter GLM-5.2 model on a laptop with 25 GB of RAM. Not fast. Not practical for production. But it ran. It produced coherent output. And it proved that the wall between "data center AI" and "your computer AI" is dissolving.

The MoE complication

Mixture of Experts models throw a wrench into the quantization story. Models like DeepSeek V3, Qwen3, and GLM-5.2 use a trick where only a fraction of the model's total parameters are active for any given token. A 671B model might only use 37B parameters per token. That means inference is fast — you're only computing with a small subset of the model.

But here's the catch: you still have to load all 671B parameters into memory. The model doesn't know which experts it'll need until it sees your input. So the memory cost is the full model, even though the compute cost is a fraction.

This makes quantization even more critical for MoE models. Without it, a 671B model needs 1.3 TB of memory at FP16. With FP8 — which is how DeepSeek ships it — that drops to about 670 GB. Still enterprise territory, but the point stands: quantization is the only reason these models are remotely deployable outside of mega-clusters.

There's also a llama.cpp trick for MoE: you can offload just the expert layers to CPU while keeping the shared attention layers on GPU. Since experts are only accessed selectively, the CPU can handle that part without becoming a massive bottleneck. This is how people run 30B+ MoE models on 12 GB consumer GPUs.

Where quantization hurts (and where it doesn't)

Not all tasks survive quantization equally. Here's the honest breakdown:

Chat and conversation? Barely affected. The model needs to be coherent and responsive, and 4-bit models are both. You'll never notice the difference talking to a Q4_K_M model versus FP16.

Summarization? Robust. The model is condensing and rephrasing, which is forgiving of small precision losses.

Creative writing? Surprisingly tolerant. Slightly less precise word choice is indistinguishable from normal creative variance. A Q4 model writing a story is fine.

Code generation? Here's where it gets dicey. Code requires syntactic precision — exact API names, correct indentation, consistent variable references. Quantization noise can erode that sharpness. A 4-bit model will write more bugs than its FP16 counterpart. Not dramatically more, but measurably. If you're using AI for production code, this matters.

Math and multi-step reasoning? Most sensitive. Precise numerical computation and logical chains amplify small errors. A wrong weight in attention can cascade through a proof or calculation. If you're doing math-heavy work, stick to INT8 or higher.

Long-context tasks? Double whammy. Quantization noise accumulates over long sequences, and the KV cache (the model's working memory for context) is usually not quantized — it stays at FP16 regardless of weight precision. So at long contexts, you're paying full memory price for the cache while dealing with accumulated quantization noise in the weights. Some newer inference engines now support INT8/FP8 KV cache, which helps, but it's not universal.

The practical decision tree

If you're trying to figure out what quantization to use, here's the no-nonsense version:

Running in production on enterprise GPUs? Use FP8. It's near-lossless, 2x smaller, and hardware-accelerated on H100+. This is what DeepSeek, Meta, and everyone serious is doing.

Running on a single consumer GPU (RTX 4090, etc.)? Use AWQ or GPTQ at 4-bit. Best quality-to-size ratio for GPU inference. You'll fit models 4x larger than you would at FP16.

Running on CPU or hybrid CPU+GPU? Use GGUF with Q4_K_M or Q5_K_M. It's the community standard, it works everywhere, and the K-quant scheme is well-optimized for mixed setups.

Have some memory headroom and care about quality? Q6_K (GGUF). About 6.6 bits per weight. You lose almost nothing compared to FP16 and gain 2.5x memory savings. This is the quality-conscious choice when you can afford it.

Desperately short on memory? Q3_K_M or Q2_K. It'll work. It won't be great. But sometimes "works" is all you need.

Why this matters beyond technical curiosity

Quantization isn't just a performance optimization. It's the technological foundation of AI sovereignty.

When a model can be compressed to fit on consumer hardware, it changes who controls AI. You don't need OpenAI's permission. You don't need a $20/month subscription that charges you per token. You don't need to send your data to someone else's server. You download a file, run it locally, and the model is yours.

That's the promise of open-source AI, and quantization is what makes it real. The gap between "a model exists with open weights" and "you can actually run it on your hardware" is bridged entirely by compression technology. Every time quantization gets better, that gap narrows further.

We're already at the point where models that rival GPT-4 class performance can run on a gaming PC. The question isn't whether local AI will be viable — it already is. The question is whether you're set up to take advantage of it.

---

If you want to use open-source AI models — quantized, optimized, running on your own machine — CopperRiver is a desktop AI assistant for Mac that handles all of this for you. No GPU management, no quantization configuration, no API keys. Just download and talk.