What runs on what

Part of the Local Host Lab notebook. To try a specific model and machine, use the calculator; for prices, the cost-performance table; for every source and date, the sources page.

Two labels are used throughout. measured here marks a number measured in this notebook, linked to the entry it came from. public marks a number someone else measured and published, cited with its source and the date of the result.

To produce one token, a model reads every weight that token uses and the attention cache built so far. For one conversation at a time this reading is the bottleneck, so single-stream decode speed is close to memory bandwidth divided by the bytes read per token. Measured runs in this notebook reached 75 to 82% of that ceiling for dense models (see the calibration table). Capacity is a hard limit instead: a model either fits in fast memory or spills to something an order of magnitude slower.

Mixture-of-experts models change the trade. They need memory for all their parameters but read only the active ones per token, so a 117B model with 5.1B active parameters can decode faster than a dense 27B. That is why large-memory, modest-bandwidth machines (a DGX Spark, a Ryzen AI Max, a Mac) are useful at all.

Each rung lists the memory and bandwidth from the vendor's specification, two derived columns, and measured decode speeds. The derived columns use the calculator's arithmetic: the largest dense model whose 4.9-bit weights (Q4_K_M) fit after reserving 5% of memory (more on unified-memory machines) and 2 GiB for cache and buffers, and the bandwidth ceiling for a 27B dense model at Q4_K_M (a 16.5 GB file). Both are estimates; the measured columns are not.

Multi-device rows show memory and bandwidth per device. Under layer split only one device reads at a time, so the ceiling uses one device's bandwidth; under tensor parallelism all devices read at once. On two V100s in this notebook, switching from layer split to tensor parallel roughly doubled decode (entry 03). Apple and AMD unified-memory rows show total memory; the GPU can address most but not all of it, so their "fits" column assumes 93% (DGX Spark, Ryzen AI Max) or 75% (Macs, the macOS default) is usable. A 16 GB card misses the 27B at Q4_K_M by a small margin; with the slightly smaller IQ4_XS file and a 4-bit KV cache it fits, as entry 13 shows.

The llama.cpp project keeps three community scoreboards (CUDA, ROCm, and Apple Metal) that run the same small model with the same command: Llama 2 7B at Q4_0, a 3.56 GiB file, generating 128 tokens from an empty context. It is an old model and a small one, so the absolute numbers are higher than anything useful today, but it is the only probe run the same way on all of this hardware. The ratio column shows what fraction of the bandwidth ceiling each result reached.

gpt-oss-120b in its released MXFP4 format is a 59.02 GiB file with 5.1B active parameters per token. It fits on one 96 GB workstation card and on every 128 GB unified-memory machine, which makes it a useful common probe for that class.

On the 128 GB machines decode falls with context: on the DGX Spark from 58.7 tok/s at an empty context to 42.8 at 32K tokens, and on the Ryzen AI Max+ 395 from 56.6 to 35.0 at 64K, in the same sources. In this notebook the DGX Spark ran larger MoE models than this one: DeepSeek-V4-Flash (284B, 13B active) at 19.2 tok/s and Qwen3.8-Flash-Next (180B, 6B active) at 43.2 tok/s with speculative decoding (entries 08 and 10).

The machines measured here are a narrow slice of the ladder: datacenter cards from 2017, one unified-memory desktop, two gaming cards and a laptop-class chip. The table repeats their headline decode numbers next to the bandwidth ceiling, so they can be read against the public figures above.

1. Builds differ between rows. Scoreboard entries are single runs submitted by different people with whatever llama.cpp commit they had. The Apple M1 to M4 rows were all run at a November 2023 build and the M5 rows at an August 2026 build; the same M2 Ultra rose from 94.3 to 125.2 tok/s on that model between those two builds.
2. Backends differ by more than hardware generations. On the same Radeon RX 7900 XTX the Vulkan backend beat ROCm. On a DGX Spark, gpt-oss-20b decoded at 49.7 tok/s under Ollama at launch and 83.4 under llama.cpp four months later.
3. Empty-context numbers are the best case. Every public number on this page is at or near an empty context. In this notebook a dense 27B fell to 38% of its short-context speed at 224K tokens.
4. Speculative decoding is left out of public rows. The scoreboards measure plain decoding. With a draft head (MTP), measured speed here rose 1.5 to 2.3 times on every machine, so a measured-here number with MTP is not comparable to a public number without it; those rows say so.