Select your hardware and quantization level to see which open-source models you can run locally, how much RAM they need, and estimated generation speed.
Model size = (total parameters × bits per parameter) ÷ 8, plus overhead for KV cache and metadata. For example, GLM-5.2 (744B MoE) at 4-bit needs roughly (744 × 4.5) ÷ 8 ≈ 415 GB of RAM — all experts must be loaded. MoE models use total parameters for memory sizing but only active parameters for speed estimation.
For dense models, tokens/sec ≈ memory_bandwidth ÷ (2 × model_size). For MoE models, we adjust the effective size based on active parameters plus shared layers (~30%). This is a simplified model — actual speeds vary by ±30% depending on the inference engine (llama.cpp, vLLM, MLX), batch size, context length, and expert routing efficiency.
LLM inference is memory-bound, not compute-bound. Generating each token requires loading the entire model from memory. This is why Apple Silicon (with 400+ GB/s unified memory) can outperform PCs with the same RAM but slower memory, and why GPUs with HBM (1000+ GB/s) are dramatically faster.
Sometimes, with CPU offloading. llama.cpp supports partial GPU offloading where some layers run on GPU and the rest on CPU. This works but is much slower (5-15 tok/s instead of 30-80). It's usable for batch processing but painful for interactive chat.
MoE models like DeepSeek V4-Pro (1.6T total, 49B active) or GLM-5.2 (744B total, 40B active) need memory for ALL parameters — every expert must be in RAM. But during inference, only the active experts are loaded per token, making them much faster than a dense model of the same total size. A 744B MoE at 4-bit needs ~415 GB of RAM but generates tokens as fast as a ~40B dense model. This is why MoE has become the dominant architecture for frontier open models.
Don't want to deal with hardware requirements? CopperRiver handles model hosting for you →