LocalLLM Rigs

What is tokens per second?

Tokens per second (tok/s) measures how quickly a language model produces output. A token is a chunk of text, often part of a word; in English, one token averages roughly three-quarters of a word. Generation speed mostly depends on memory bandwidth, because the model's weights are read again for each new token. In this site's catalog, speeds on a 70B model range from 0 tok/s to 14 tok/s. The Apple Mac mini (M4, 16GB) shows 0 because the model doesn't fit in its memory. The Apple Mac Studio (M3 Ultra) reaches 14 tok/s with 819 GB/s of bandwidth. Most 128 GB machines sit between 4.5 and 9 tok/s. Generation speed is different from prompt processing, the time spent reading your input before the first word appears. Several Macs here trail high-end NVIDIA GPUs on prompt processing.