Local Host Lab · V100 · 05

A 188 GB model on two 32 GB cards

Qwen3.8-Flash-Next at Q8_0 is a 188 GB file, three times the memory of the two cards it ran on. Three ways of running it were compared on the same file. The winner keeps the model's error within 0.0024 KL of llama.cpp and decodes at 55 to 58 tokens per second, 2.6 times the tuned llama.cpp setup.

October 2026 · Qwen3.8-Flash-Next Q8_0 (188.2 GB, Unsloth GGUF) · one node, 2 × Tesla V100-PCIE-32GB, no NVLink, 377 GB RAM · llama.cpp, Strata v0.1.42

Note 02 put a 27B model on these cards and found that the second card bought capacity, not speed. This model is different in kind: a mixture-of-experts whose weights are three times the memory of both cards together. Most of it has to live in system RAM, and the question becomes which parts sit on the GPUs and how the rest gets there.

The rule was fixed before any run. If Strata on two cards came out about as precise as Q8_0 on llama.cpp and much faster, it would be the serving backend; otherwise llama.cpp at Q8_0 would be.

The setup

One borrowed HPC node: two Tesla V100-PCIE-32GB with no NVLink, two Xeon Gold 6248 (20 cores each, AVX-512 without VBMI), 377 GB of RAM and a locked-memory limit of about 47 GiB. Compute nodes have no outbound network, and the weights sit on a shared filesystem.

The weights are Unsloth's Q8_0 GGUF of Qwen3.8-Flash-Next (GGUF builds), 188.2 GB in six shards. From the file's own metadata: 48 layers, of which 12 are attention layers and the rest linear attention; two KV heads of 256 dimensions; 24,576 routed experts; a native context of 262,144 tokens. The file also carries a 54 GB per-layer lookup table, from which each token reads 16 rows.

Q8_0 was chosen because it is the highest precision that fits in one node's RAM, and the highest that Strata v0.1.42 will load, so all three arms ran the same file.

Speed was measured with a long math TeX prompt, temperature 0.6, thinking on, 512 output tokens, single stream. Single runs unless a row says otherwise.

Tuning llama.cpp from 9.5 to 22

The cause of decode jumping between 6 and 24 tok/s was the lookup table being read lazily. By default llama.cpp maps tensors larger than 4 GiB on demand. Once the table's pages fall out of the file cache, every token reads its rows from the shared filesystem. With --lazy-mode off the table is loaded into RAM: decode on the same configuration went from 8.0 to 18.6 tok/s, the run-to-run spread disappeared, and MTP stopped slowing down prompts. Rows t1 to t8 were all affected.

• Without NUMA interleave the whole model lands on one socket, and prefill swung between 120 and 293 tok/s on identical runs. numactl --interleave=all made it stable.
• MTP in llama.cpp makes prompt processing about 2.5 times slower (a 4,096-token chunk went from 13 s to 36 s). The cause was not found.
• The tensor split, with experts in RAM, is slower than the default: about 10 tok/s decode.

Three ways to use the second card

Strata has no tensor parallelism. On two cards it offers a layer split, where each card holds half the layers, or a mode where the second card holds nothing but more experts (--expert-cache-device1 auto --remote-expert-opt). Both were measured. The second mode was re-measured on a healthy node, as the next sections explain:

Batching four streams lowers total throughput in this mode, because the prompt reads serialise, so the server runs one stream per node. With one card, 4,864 of the 24,576 experts (19.8%) fit in VRAM, and on math text they served about 80% of the routed calls.

Precision

23 math TeX requests, teacher-forced, 16,031 positions. The reference is the same Q8_0 file on llama.cpp's CPU backend with fp32 accumulation. The metric is KL from the reference to each arm over the reference's top 256 tokens plus one bucket for the rest.

Paired over the 23 requests, against llama.cpp on the GPUs:

Both Strata configurations with a resident RAM budget add about 0.021 KL, roughly the distance between two adjacent quantisation levels. The expert-cache mode cannot be combined with that budget, and it adds 0.0024. Its KL interval excludes zero; its top-1 interval does not. The largest 1% of positions carry 47% of its total KL, against 46% for llama.cpp on the GPUs, and its single worst position is 8.7 nats against 4.7.

The node that printed 3333

The first precision round made Strata's two-card modes look worse than one card: the expert-cache arm at 0.0410 KL, and a layer split whose per-position output was NaN at 11,749 of 16,031 positions.

All of those runs were on one node. On that node, llama.cpp's serving configuration on both cards answered a short greedy sanity question with 3333... every time, while the identical binary, model and flags answered correctly on two other nodes with the same driver, VBIOS and kernel. Each card on its own was correct. A peer-to-peer test (peer copies with and without peer access, plus direct peer reads, 256 MB three times) found zero mismatches. The cause was not found, and the node was released.

Re-measured on a healthy node, the expert-cache arm dropped from 0.0410 to 0.0170 and the layer split had no NaN rows. From then on every new node got the two-question sanity check before any measurement.

Where the 64 GiB goes

A production instance fills both cards: 31.4 GiB each, 62.8 of 64 GiB in all. Strata places what it needs first and fills the rest with experts.

About 10 GiB is required; the other 51 GiB or so is cache. Together the two cards hold 10,523 of the 24,576 experts, 43%. Of the cache on the first card, 5.1 GiB is lent to prefill while a long prompt is read. Every expert in VRAM is one that does not cross PCIe from RAM for each token that routes to it.

Long context

At a 131,072-token maximum, KV int8, reading an unseen prompt from scratch and generating 512 tokens, two runs per length:

In a real math-agent run (487 requests, prompts up to 97.7K tokens, 1K to 17K tokens generated each), the engine logs show decode barely moving with length: 48, 51, 47 and 46 tok/s for prompts under 16K, 16K to 32K, 32K to 64K and over 64K.

The full 262,144 tokens

The KV cache is small. At int8 (int8 codes plus one fp16 scale per 64 values) it measured about 15.6 KB per token from free memory, because only 12 of 48 layers have attention and each has two KV heads. Going from 131,072 to 262,144 cost 1.9 GiB on the first card and moved its expert slots from 4,619 to 4,226.

A 223,076-token prompt read at 585 tok/s (382 s) and then decoded at 41.7 tok/s.

The first two attempts at 223K to 245K tokens stalled between 66K and 82K and were killed by the engine's 300-second watchdog. Strata's default reads the 54 GB lookup table row by row with direct I/O, bypassing the page cache; cold, that is about 2,500 rows per second against 16 rows per token. Loading the table into locked RAM needs a 54 GB lock, over the 47 GiB limit. The fix was --ple-io mmap plus one sequential read of the table's shard at startup, 19 seconds from the file cache.

Prefix cache and routing

During that agent run, three nodes sat behind a small local proxy. Prefill took 25% of engine time (8,406 of 33,428 s), and 6,527 s of it, 78%, was 107 long prompts read from scratch. The proxy sent each request to the least busy node, so one conversation hopped between nodes and lost its prefix cache every time it moved. Follow-up requests that stayed on the same node read only 36 to 661 new tokens.

The proxy now routes by conversation: the key is a hash of the first two messages, and a new conversation goes to the node with the fewest conversations plus requests in flight.

Smaller experts

Later, the production instance slowed to about 4 tok/s decode, and long prompts stalled at the first layer. That led to two findings: what caused it, and that the experts can be stored at lower precision without a measurable cost.

The lookup table was being evicted by the expert load itself. The engine reads the experts through the file cache, and on this node the file cache never grew past about 100 to 150 GB. Streaming 92 to 120 GiB of experts through it pushed out the table pages that had been warmed before startup (Q8_0: 100% cached before, 0% after). Reading the table's shard again after the engine finished loading (14 to 21 s) brought it back to 100%, and Q8_0 returned to its original speed: 400 / 59.8 tok/s prefill / decode at 4K, against 394 / 55.0 first measured.

Unsloth publishes the same model with lower-precision experts, all loadable by Strata. Same 23 requests, same engine arguments as the Q8_0 expert-cache arm:

Speeds are single runs at 262K context with the table re-read after loading. Q6 and Q5 cannot be told apart from Q8_0. Q5 uses 28 GiB less RAM, fits about 31% more experts on the cards, and decodes 15 to 25% faster. Q4 is a significant loss: about 0.021 KL was taken above as one quantisation level, which puts Q4 at roughly 0.4 of a level (that scale comes from community numbers, not from these runs).

Two 16 GB cards, emulated

The 16 GB V100 is common too. There were no 16 GB cards to test on, so the same node emulated them: before each start, a placeholder process allocated 16,608 MiB on each 32 GB card, leaving about 16,160 MiB, the usable memory of a 16 GB card. The table was re-read after loading, and the sanity check passed before and after every configuration.

• Best setting: UD-Q5_K_XL, second card as expert cache, full 262K context. Precision indistinguishable from Q8_0, decode 35 to 40 tok/s, about 60% of the same quant on the 32 GB cards.
• Cutting the context to 128K, or keeping part of the KV cache in RAM, frees room for experts and raises the hit rate, but adds only 1 to 2 tok/s.
• A layer split doubles long-prompt prefill (1,571 against 784 tok/s at 116K) and loses about 20% of decode. Its precision was not measured on a healthy node.
• Q4 is about 30% faster again, at the precision cost above.

What this settles, and what it does not

• When a mixture-of-experts does not fit in VRAM, the second card is worth more as an expert cache than as half a layer split. At 16K: one card 18.0 tok/s, layer split 26.7, expert cache 57.9. This does not contradict note 02. There a dense model was layer-split; here the second card reduces the expert bytes that cross PCIe for every token.
• Engine configuration on one file can cost as much precision as a quantisation level. With a resident RAM budget Strata added about 0.021 KL; without one, 0.0024. Precision has to be measured per configuration, not per engine.
• llama.cpp's speed problems here were I/O, not arithmetic. Lazy reads of a large table and NUMA placement mattered more than any kernel setting.
• Under agent traffic, route by conversation. Least-busy routing spent three quarters of prefill re-reading prefixes that were already cached elsewhere.
• Warm a large memory-mapped table after the weights have loaded, not before. Loading the experts evicted it and cost an order of magnitude in decode.

Not measured: what Q8_0 costs against BF16, precision on token-by-token generation and at long context, accumulated error from the int8 KV cache, anything beyond 262,144 tokens, and real 16 GB cards rather than emulated ones.

Previous: 720p video in five minutes · All V100 notes · Local Host Lab