Stop paying per token. Run open-source AI models on your own hardware — full privacy, zero API costs, complete control. Here's exactly how to do it.
Every time you send a prompt to ChatGPT or Claude, your data leaves your computer, travels to someone else's server, and gets processed on hardware you don't control. You pay per token, you're subject to rate limits, and the provider can change the model, the price, or the privacy policy whenever they want.
Running AI locally flips that equation. The model lives on your machine. Your data never leaves. There are no per-token fees, no rate limits, no outages dependent on someone else's infrastructure.
Your prompts, documents, and code never leave your machine. Critical for sensitive data and regulated industries.
No API bills. Run millions of inferences — you pay for electricity, not tokens.
No 429 errors, no usage caps, no queueing. Your hardware, your rules.
Choose the model, the quantization, the context length. Nobody silently changes the model on you.
RAM is the bottleneck, not CPU speed. Models live in memory, and bigger models need more of it. Quantization (Q4, Q8) compresses models at a small quality cost.
| Tier | RAM | Speed |
|---|---|---|
| Minimum | 8 GB | 5-10 t/s (CPU) |
| Recommended | 16 GB | 10-25 t/s |
| Enthusiast | 32 GB+ | 25-60 t/s |
| Server / GPU | 64 GB+ VRAM | 60-200+ t/s |
Quantization reduces model precision to save memory. Q4 (4-bit) uses roughly 25% of the original size with minimal quality loss. Q8 (8-bit) uses about 50% with almost no quality loss. FP16 (16-bit) is full precision. For most use cases, Q4 is the sweet spot — you get 90%+ of the model's intelligence at a quarter of the memory cost.
Pick the biggest model your hardware can handle. Bigger isn't always better for every task, but it's usually better overall.
| Model | Params | Best For |
|---|---|---|
| Qwen 3.6 (4B) | 4B | Lightweight chat, basic coding |
| Phi-4 | 14B | Reasoning, math, small footprint |
| GLM-5.2 | 110B (MoE) | General purpose, tool calling, agents |
| DeepSeek V4 | 671B (MoE) | Coding, reasoning, cost-efficient |
| MiniMax M3 | 230B (MoE) | Long context, multilingual |
| Kimi K3 | 2.8T (MoE) | Frontier open-source, complex reasoning |
Note: MoE (Mixture of Experts) models activate fewer parameters during inference, making them faster and cheaper to run than dense models of the same total size.
| Tool | Type | Setup |
|---|---|---|
| Ollama | CLI / API | Easy |
| llama.cpp | Library / CLI | Medium |
| LM Studio | GUI App | Easy |
| vLLM | Server | Advanced |
| CopperRiver | Desktop App | Zero setup |
The fastest way to get started from the terminal.
# Install Ollama (macOS)
brew install ollama
# Start the server
ollama serve
# In another terminal, run a model
ollama run qwen3:4b
# Or run a larger model
ollama run glm5.2
# Use as an OpenAI-compatible API
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "qwen3:4b", "messages": [{"role": "user", "content": "Hello!"}]}'If you'd rather not touch the terminal, CopperRiver bundles everything — models, browser, terminal, file access, automation — in a desktop app.
1. Download CopperRiver for Mac 2. Pick a model (GLM, DeepSeek, Qwen, MiniMax, or Kimi) 3. Start chatting # That's it. No command line, no config files, no GPU setup. # You also get: web browsing, terminal access, file reading, # scheduled automations, and agent capabilities — all built in.
For serving models to multiple users or running agents at scale.
# Install vLLM (requires GPU)
pip install vllm
# Serve a model with OpenAI-compatible API
python -m vllm.entrypoints.openai.api_server \
--model TheBloke/glm-5.2-AWQ \
--quantization awq \
--tensor-parallel-size 1 \
--port 8000
# Now any OpenAI-compatible client can use it
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
response = client.chat.completions.create(
model="TheBloke/glm-5.2-AWQ",
messages=[{"role": "user", "content": "Hello!"}]
)Chat with AI about sensitive documents, code, or personal data — nothing leaves your machine.
Get inline code suggestions while you program, without sending your codebase to a third party.
Drop PDFs, contracts, and spreadsheets into a local model. Ask questions, get summaries.
Build agents that browse, run commands, and process files using local models. CopperRiver does this out of the box.
Yes. With 8GB+ RAM you can run small models. For larger models, 16-32GB is recommended. Tools like Ollama and CopperRiver make setup easy — no GPU required for quantized models.
CopperRiver is the easiest — it's a desktop app with models pre-configured, plus browser, terminal, and file access. For CLI users, Ollama is simplest: 'ollama run qwen3:4b'.
No. CPUs can run quantized models at 5-20 tokens per second. GPUs are faster but not required for most use cases.
For 8GB RAM: Qwen 3.6 (4B), Phi-4. For 16GB: GLM-5.2 (quantized), DeepSeek V4 (quantized). For 32GB+: full-size models. CopperRiver bundles GLM, DeepSeek, Qwen, MiniMax, and Kimi.
The models and software are free and open-source. You pay for hardware and electricity. CopperRiver offers managed model hosting and agent capabilities from $9/month.