Memory bandwidth explained
Memory bandwidth is the rate at which a processor can read from and write to its RAM, measured in GB/s. When an LLM generates text, it streams essentially all of its active weights from memory for every token. Because of this, bandwidth usually caps tokens per second more than raw compute does. A rough ceiling is bandwidth divided by model size. For example, the Apple Mac Studio (M3 Ultra) has 819 GB/s, and a roughly 42 GB 4-bit 70B model gives a ceiling near 19 tok/s; the catalog lists 14 tok/s. In this catalog, bandwidth ranges from 120 GB/s to 819 GB/s. Two machines with equal memory capacity can perform very differently. Both the GMKtec EVO-X2 and the Mac Studio (M4 Max) have 128 GB, but their bandwidths are 256 GB/s and 546 GB/s.