LocalLLM Rigs

Open-weight models explained

An open-weight model is a language model whose trained parameters, called weights, are published for download. This lets you run it locally instead of through a cloud service. Familiar families include Llama, Mistral, Qwen and Gemma. "Open-weight" is not the same as open source: the training data and code may stay private, and licenses can restrict commercial use. Always read the license. Model size is counted in billions of parameters (B) and decides which hardware can run it. A 70B model quantized to about 4 bits needs roughly 40 GB of memory. That fits on the Apple Mac mini (M4 Pro, 64GB), which runs it at about 5.5 tok/s. The Apple Mac mini (M4, 16GB) is limited to small models and lists 0 tok/s for 70B. Very large models, such as 235B at Q4, need more memory than a single Minisforum MS-S1 Max provides.