Serving modern models on 2017 silicon

One exchange, verbatim, from the middle of these notes:

Every word of the first half was true. It took another five turns, a twenty-minute answer that would not converge and one added sentence that stopped it dead before the model accepted where it was — and an earlier run of the same weights, in another language, never accepted it at all. That is note 01.

Volta is supposed to be finished. It has no BF16, no FP8, no FlashAttention and no future in PyTorch past 2.11; modern inference servers refuse to load on it outright. It also sits, in quantity, in every academic cluster that bought GPUs before 2019 — which makes "what still runs on it" a practical question rather than a nostalgic one.

Four experiments, each pushing a different limit: the largest weights, the longest context, the hardest numerics, the most expensive output.

Nine things that transfer

1. The barrier is the ecosystem, not the arithmetic

Modern inference servers refuse sm_70 and need community patches to load at all. FlashAttention will never support Volta — the tensor core shape is wrong and the maintainers closed the question in 2024 — which also eliminates everything built on it. One major kernel library dropped V100 outright in 2025. Meanwhile llama.cpp and ComfyUI both work fine. When someone asks whether a V100 can run X, check whether anyone compiled a numerical path for it, then check memory, and only then doubt the FLOPs.

2. Card count is a software problem

A quota of two GPUs per node does not mean two GPUs maximum. Running one RPC worker per card and a head process that owns no card of its own turns four nodes into a single eight-card device pool, and that is what made 202 GB of weights loadable. The ceiling on an old cluster is whether you can staple cards together.

3. Memory layout usually beats numerical cleverness

In the video project, every dynamic-range fix put together was worth less than one observation: that 39.3% of the parameters consume only the timestep and can be precomputed into a table. That was 3.75×. The next largest win, 1.65×, came from noticing that a pipeline split at batch size 1 leaves half the machine idle. Neither is an arithmetic optimisation.

4. Time the whole wall clock before optimising any of it

The same project made its denoise loop 1.65× faster and then discovered the loop was 22% of the job. The largest single item was 37.8 minutes of offload-wrapping for a text encoder used for eleven seconds — and that number was really 63 GB crossing a shared filesystem, varying 7.6× between runs of identical code. Caching two constants took cold start from 25 minutes to under two, with a bit-identical output.

5. Benchmarks with token caps hide convergence damage

At one bit, a 744B model lost 0.5 to 2 points on standard benchmarks — inside the sampling noise. In open-ended conversation it wrote for twenty minutes and could not stop, and one added sentence made it 9.5× faster at the same quality. Every benchmark item has an output cap, and the cap is exactly what hides that symptom. To measure what quantisation costs, include one open-ended task with no cap.

6. A model's account of its own deployment is a guess, and it stays a guess after you tell it

Turn 2 of that conversation asked the 744B model what hardware it was running on. It reasoned its way to the modal answer for something of its size, four to eight A100s or H100s behind a production inference server, and then volunteered:

"I could be running on a single H100 or a rack of older V100s
and I'd have no way to tell you the difference."

It was, at that moment, running on a rack of older V100s. Told so in the next turn, it declined to accept it and filed its own hardware as a thought experiment. What eventually convinced it was not being told again, but the twenty-minute runaway above and the single sentence that stopped it:

"Yes, I believe it now. I didn't before because I pattern-matched
'extraordinary claim → probably fictional' [...] the behavioural
evidence in this conversation is the strongest possible proof I could
have from the inside."

Nothing the model said about its own deployment carried information, in either direction. Not the confident guess, not the refusal, not the agreement. An earlier run of the same weights on the same cards, in another language, is the control, and it went the other way: it called the truth a what-if game, never conceded at all, and predicted the opposite capability would break first (its four quotes are in note 01). What the two runs do agree on, to within 0.7%, is how long the runaway lasted. The self-report varies run to run; the behaviour does not. That generalises well past hardware, to every question of the form "are you being served the way you think you are": the only instrument that worked here was behaviour under a task with no cap on it. Both endings are in note 01.

7. Speculative decoding is the free lunch on bandwidth-bound decode

A self-speculative draft head returned a 1.53× median (1.96× on code) for 3 GB of memory and 10% of prefill. But acceptance rates swing thirty points between runs at the same settings, so a single A/B is not enough to call one machine better than another at speculation. Report the median of several runs, or report nothing.

8. Static sparse attention is dead on video diffusion, and it is worth measuring rather than assuming

The sparsity is real: the best 940 of 19,300 keys carry a median 89% of the attention mass. The locality is not: the obvious "own frame plus non-video" window captures 10.3%, and zero heads of 56 pass. Finding the right 940 requires the full score computation that sparsity exists to avoid. That is the whole difference between static and dynamic sparsity, and it costs one afternoon to establish.

9. "Can it run" and "can you afford it" are different questions

On identical cards, the 33B video model needs about 3.1 hours for a ten-second clip while a distilled 22B produces five seconds at 1920×1088 in ten minutes.

On old hardware, distillation is worth more than every kernel optimisation combined — and the heavyweight is still the right project when the question is what the hardware can do at all.

The baseline these grew out of

Before any of the above, the same two cards ran a 27B model at three quantisation levels on identical items: Q6_K at 25.0 tok/s, Q8_0 at 23.1, and full BF16 — which fits, at 54 GB across two cards — at 14.8, with prefill at 564 tok/s. A single card ran a 32B model at 22.3 tok/s through a community fork of a modern inference server, whose entire reason for existing is that it lowers the compute-capability floor to 70.

Two conclusions from that round survived everything since. Without NVLink, a second card solves "it fits", not "it is fast": under layer splitting, single-stream speed is about that of one card. And a reasoning model on slow hardware compounds: a thousand tokens of thinking at 22 tok/s is a minute per turn, so a good benchmark number is not the same thing as a usable session.