LocalLLM Rigs

What is generation speed?

Generation speed, also called decode speed, is how fast a model produces output tokens after it has read your prompt. It is measured in tokens per second (tok/s). A token is roughly a short word or part of a word. Each new token requires reading the model weights from memory, so generation speed depends mainly on memory bandwidth. Our catalog shows this pattern for 70B models. The Mac Studio (M3 Ultra) has 819 GB/s and reaches 14 tok/s. The Mac Studio (M4 Max) has 546 GB/s and reaches 9 tok/s. The NVIDIA DGX Spark has 273 GB/s and reaches 4.5 tok/s. The Mac mini (M4, 16GB) shows 0 tok/s because a 70B model does not fit in its memory. Generation speed is a different measure from prompt processing speed, which is how fast the input is read. Prompt processing depends more on compute.