LocalLLM Rigs

What is tok/s?

tok/s means tokens per second, the standard measure of how fast a local model generates output. A token is a chunk of text, often part of a word, so an English word averages a little more than one token. This site's spec tables list generation speed on a 70B model. Figures range from 0 tok/s on the Apple Mac mini (M4, 16GB), which cannot load the model, through 4.5–5.5 tok/s on most 128 GB and 64 GB machines, up to 14 tok/s on the Apple Mac Studio (M3 Ultra). For comparison, roughly 5 tok/s is usable for chat, though slower than comfortable reading speed. Results depend on model size, quantization and context length. Prompt processing, the speed of reading your input, is measured separately and is often much higher.