LocalLLM Rigs

KV cache explained

The KV (key-value) cache is memory an LLM uses to store attention keys and values for every token already in the context. Because of the cache, the model does not recompute the whole conversation for each new token. The cache grows linearly with context length. It needs memory on top of the model weights. For example, Llama 3 70B uses about 0.3 MB per token at 16-bit precision. That comes to roughly 2.5 GB at an 8K context and about 10 GB at 32K. This is why the size of a model file alone does not tell you whether it will run comfortably. The Mac mini (M4 Pro, 64GB) fits 70B-class quants of around 40 GB. However, long contexts eat into the remaining memory. Many runners can quantize the KV cache, for example to 8-bit, to reduce its size.