LocalLLM Rigs

What is ROCm?

ROCm is AMD's open-source platform for running general-purpose compute work, including AI inference and training, on AMD graphics hardware. It is AMD's counterpart to NVIDIA's CUDA. Frameworks such as PyTorch and inference engines such as llama.cpp can use it, although some tools are built for CUDA first. ROCm matters for machines built on the AMD Ryzen AI Max+ 395, such as the GMKtec EVO-X2 and the Minisforum MS-S1 Max. Both offer 256 GB/s of memory bandwidth and list about 5 tok/s on a 70B model. Support for a given consumer chip depends on the ROCm version, so check AMD's compatibility list and your tool's documentation before you buy. Some runtimes also offer a Vulkan backend as an alternative. Apple Macs don't use ROCm or CUDA; they rely on Apple's Metal instead.