Local Host Lab

V4-Flash is a 284B mixture of experts with 13B active. On one node with the experts in CPU memory it decoded at 4.75 tok/s, three to five times below the estimate made beforehand. The CPU side used about 17 GB/s of the machine's memory bandwidth, and the experts were stored as MXFP4, whose CPU path in ggml is less mature than Q4_K's.

Joining four to six cards over RPC crashed llama.cpp: rpc_server::graph_compute rejected view nodes that did not belong to a local buffer. Skipping those nodes, while keeping the existing security check intact, fixed it. After the fix, IQ3_XXS on five cards ran prefill at 259.7 and decode at 21.2 tok/s, 12.5 times and 4.5 times the single node. Q4_K_XL (144.4 GiB) fits on six cards with about 46 GiB left for the KV cache. Five cards were slightly faster than six; the sixth card buys KV room, not speed.

Two PCIe cards cannot say what NVLink would add, so a 4 × V100-SXM2 machine was rented by the hour. Instead of comparing two machines, the interconnect was switched on and off on the same one (NCCL default against NCCL_P2P_DISABLE=1), with layer split as a zero control and a card pair that has no NVLink as a negative control.

The gain is all in prefill: +20 to 30% on two cards, +55% on three fully linked cards, +76% on four. Decode gains 1.4 to 4.1% without speculation and nothing, or slightly less than nothing, with MTP. llama.cpp's own all-reduce does not use NVLink at all; without NCCL at build time the link is wasted. NCCL_P2P_DISABLE is a lower bound for PCIe, not a real PCIe machine, so the two comparisons are reported separately.

UD-IQ3_XXS, 104.2 GB (97.1 GiB); about 18% of tokens take a different argmax from the original. The larger quant often recommended for 128 GB machines does not fit in 119 GiB. Loading has to bypass mmap, because on unified memory the page cache and the CUDA buffer each hold a copy and the load crashed at 2 minutes 50 seconds; without mmap it loaded in 93 seconds.

Decode is 19.2 tok/s and prefill about 400. Six A/B changes (KV precision, ubatch, newer builds) all stayed within ±5%: the bottleneck is the overhead of many small operators across 43 layers, not bandwidth, of which about 30% is used. The KV cache is 3.68 KiB per token, so a million tokens of context is 3.7 GiB.

Speculative decoding sped up code by 79% and prose by 0%, and it conflicts with context checkpoints. An agent rolls back a little every turn, so with speculation on the second turn recomputed 15,600 tokens: 51.7 seconds for the turn, against 7.1 seconds without it. Keeping whole sessions' KV in host memory brought switching back to a 125K-token session from 532 seconds to 0.41. A second Spark joined to the first would decode about 30% slower single-stream.

A clean bandwidth-bound case (82% bandwidth efficiency), so speed tracks weight size closely.

MTP took the baseline from 7.87 to 18.27 tok/s (2.32×); at a draft depth of 4 acceptance fell to 24%. NVFP4 is 15% faster than Q4_K_M at a similar size but has 4 to 5 times its KL divergence. Measuring on English alone understates the loss: KL on Chinese and code was consistently about 30% higher. With speculation on, smaller weights are not always faster, because quantisation error lowers the acceptance rate.

180B total, 6B active per token, including a 51B lookup table that llama.cpp reads from NVMe on demand, so it fits in 128 GB. UD-Q4_K_XL with MTP decodes at 43.2 tok/s with 80.6% acceptance; prefill, at 123 tok/s, is the weak side.

It served as the backend for a development instance of an Anthropic Messages endpoint for NCSA Lumen, the university's open-weight model service, so that Claude Code and the Anthropic SDK can call Lumen-hosted models. The end-to-end smoke test passed 9 of 9. Rewriting mid-conversation system messages in place, instead of moving them to the front, kept the backend's prefix cache valid and took per-turn prefill from about 40,000 tokens and 73 seconds to about 100 tokens and 0.5 seconds. One trap: llama.cpp divides -c evenly across parallel slots, so 65,536 with two slots gave each request 32K, while Claude Code's system prompt alone is 26K; --kv-unified fixes it. The client side is claude-code-lumen.

Qwen3.5-122B-A10B with INT4 weights and MTP-2 under vLLM: 37 to 40 tok/s on long thinking outputs, MMLU 90.0% and GSM8K 96.0% (n = 100). NVFP4 on the same machine ran at 16.6 tok/s, half the INT4 speed, because the FP4 kernels were written for SM120 and GB10 is SM121.

MiniMax-H3 at 960×544, 124 frames and 20 steps took about 410 seconds with community int8 weights, against 3.1 hours for a 10-second 544p clip on the V100s. An eight-segment, 43-second short was rendered on it.

An Ampere card (sm_86, no native FP8 or FP4) modified to 20 GB, about 19.6 GiB usable. Qwen3.6-27B, fully on the card:

MTP costs about 0.6 GB. Adding n-gram drafting on top speeds code tasks by another 8 to 13% at no memory cost. The trade is 4-bit, 72 tok/s and 128K against 5-bit, 33 tok/s and 96K. Decode barely changes with context because it is bound by reading weights.

The same card also trains small models. For a self-study course on training agents with SFT and RL, Qwen3-1.7B with LoRA went from 0.00 to 0.83 test success after 24 SFT steps (207 trajectories, about 100 seconds). GRPO from a 32-trajectory SFT start went from 0.31 to 0.64 in 40 steps and 29 minutes; from the 207-trajectory start it barely moved (0.83 to 0.84).

Qwen3.6-27B IQ4_XS fits with a 48K context and a 4-bit KV cache, leaving 335 MiB free, and decodes at about 49 tok/s. Q4_K_M does not fit; with 59 of 65 layers on the card it falls to about 20 tok/s. With K at q8_0 and V at q4_0 there is no fast attention kernel for this model's head size of 256, and prefill at 16K fell from 1032 to 42 tok/s; keeping K and V at the same 4-bit type avoids it.

The same 33B video model as entry 05, run with antirez's C and Metal engine h3.c. The M4 has native bf16, so none of the V100 numerical problems apply; the limit becomes arithmetic. A 5.17-second clip at 960×544, 124 frames and 20 steps takes 19 minutes.

bf16 GEMM tops out at about 14.7 TFLOPS whoever writes the kernel: MPSGraph reached 14.70, and a hand-written kernel swept over about 50 configurations reached 14.66. The M4 GPU has no neural accelerators; forcing the Metal 4 tensor API compiles and runs about 10% slower. Upstream's hand-written kernels are enabled only on M5 devices, so published M5 numbers describe code the M4 never runs.

At production length the output is not reproducible: two runs with the same binary, settings and seed had a mean SSIM of 0.831 across 124 frames. Timing is: the two denoise times differed by 0.01%. Correctness checks for video diffusion cannot ask for a matching full clip.