LocalLLM Rigs

What does 4-bit mean?

4-bit refers to quantization, which stores each model weight using about 4 bits instead of the 16 bits typical of the original release. This cuts memory use to roughly a quarter. Generation also gets faster, because less data has to stream from memory for each token. The cost is a small loss in output quality that is usually acceptable. A 70B model needs about 140 GB at 16-bit but around 40 GB at 4-bit. That is why 64 GB machines like the Apple Mac mini (M4 Pro, 64GB) can run 70B-class models. Common 4-bit formats include GGUF's Q4_K_M, which averages a bit under 5 bits per weight because some layers are kept at higher precision. On this site, 70B speed figures assume a 4-bit-class quantization.