LocalLLM Rigs

What are model weights?

Weights, also called parameters, are the numbers a model learns during training. Together they are the model, and running it means loading them into memory. Their size is the parameter count multiplied by the bytes per weight. At 16-bit (2 bytes), a 70B model is about 140 GB. At 4-bit quantization it is roughly 35–43 GB, depending on the format. The weights must fit in the GPU-accessible memory with room left for context. That is why the Apple Mac mini (M4, 16GB) is limited to small models, while 128 GB machines like the NVIDIA DGX Spark can hold 70B-class models. Weight size also drives speed. Each generated token reads the active weights from memory, so a smaller file on the same memory bandwidth produces more tokens per second.