Size the rig to the model, not the spec sheet
Start with the open-weight models you actually plan to run, then work backwards to hardware. Parameter count and quantization level set the memory floor. If the weights don't fit, nothing else on the spec sheet matters.
Most people running models at home use Q4_K_M, a 4-bit quantization that keeps quality close to the original while cutting size by roughly three-quarters versus FP16. At that level the numbers are simple to remember:
| Model size | Q4_K_M (weights only) | Q8_0 (approx.) | FP16 (approx.) | Typical local use |
|---|---|---|---|---|
| 8B | ~5 GB | ~8.5 GB | ~16 GB | Chat, autocomplete, quick coding help |
| 32B | ~20 GB | ~34 GB | ~64 GB | Serious coding assistants, reasoning, drafting |
| 70B | ~40–43 GB | ~75 GB | ~140 GB | Near-frontier chat, complex analysis |
These figures cover weights only. Every token in the context window also needs room in the KV cache, and that grows linearly with context length. A long context window adds several GB, and on a 70B model it can pass 10 GB. So a 32B model at ~20 GB fits a 24 GB GPU for everyday chat, but not once you ask it to read a large codebase. It loads, then runs out of room or falls back to slower system RAM.
Plan for weights + KV cache + 15–20% spare. On Apple silicon and Strix Halo, the OS and other apps share the same unified memory pool, so the whole figure is never available to the model. A 64 GB machine is a comfortable 70B Q4_K_M home. A 48 GB one is a squeeze.
Where the platforms split
8B–32B covers most home use. Coding assistants and everyday chat run well in that range, and nearly every rig in this comparison handles it. A 24 GB discrete GPU, a 32 GB-plus Mac or a Strix Halo box will all load a 32B model at Q4_K_M with sensible context. At this size you're choosing on speed, noise and price, not on whether the model fits.
70B and larger is where the field divides. At roughly 40–43 GB before context, a 70B model won't fit on a single 24 GB card. You'd need two GPUs or have to offload layers to system RAM, which slows generation sharply. The large unified memory pools of higher-memory Macs and 128 GB Strix Halo systems hold it in one piece. Above 70B, such as 100B-plus mixture-of-experts models, only the largest unified memory configurations stay in play.
So the first question is "what's the biggest model I'll run every week?", not "what's fastest?" Buy the memory capacity that answers it, then compare speed among the rigs that pass.
The mistake most buyers make: capacity without bandwidth
A spec sheet that says "128 GB unified memory" tells you a 70B model will load. It doesn't tell you whether it will run fast enough to hold a conversation. That second question is answered by memory bandwidth, and it's where many first-time buyers get caught out.
During generation, the chip streams essentially the entire set of weights from memory for every token it produces. That gives you a simple rule of thumb: memory bandwidth ÷ model size in memory ≈ the ceiling on tokens per second (tok/s). Real-world speeds land below that ceiling, never above it.
| Model at Q4_K_M (approx. size) | Strix Halo (~256 GB/s) | M3 Ultra (~819 GB/s) | RTX 4090 (~1,008 GB/s, 24 GB) |
|---|---|---|---|
| 8B (~5 GB) | ~50 tok/s ceiling | ~160 tok/s ceiling | ~200 tok/s ceiling |
| 32B (~20 GB) | ~13 tok/s ceiling | ~40 tok/s ceiling | ~50 tok/s ceiling |
| 70B (~40 GB) | ~6 tok/s ceiling | ~20 tok/s ceiling | Doesn't fit in 24 GB |
That 70B row is the decision point. On Strix Halo, a 40 GB 70B model tops out around 6 tok/s, and real output lands below that, which feels sluggish for back-and-forth chat. An M3 Ultra's ceiling for the same file is about 20 tok/s, and roughly 15–20 tok/s in practice is comfortable. Both machines have the capacity. Only one has the bandwidth to make 70B pleasant to use. Strix Halo is far better suited to 32B-class models, where its ~13 tok/s ceiling is still workable for chat.
The other half: prefill
Bandwidth governs output. Prompt processing (prefill) works differently. Reading your input is compute-bound, because the chip handles all the prompt tokens in parallel rather than streaming weights once per token. NVIDIA GPUs have far more raw compute for this step. As a result, Macs and Strix Halo show a noticeably longer time to first token (TTFT) when the prompt is long, even when their tok/s figures look respectable.
For short chat prompts you'll barely notice the gap. If you paste in a 30-page contract or a large chunk of a repository, the wait before the first word appears can rival the time spent generating a short answer.
Work out which way your traffic flows. If your typical session is long documents or codebases in, short answers out, prefill speed matters as much as tok/s. That favours a discrete NVIDIA GPU, provided the model fits in its VRAM. If you mostly chat or generate long outputs from short prompts, prioritise bandwidth and unified memory capacity instead.















