V100 notes · 01

A 744B model at one bit, on eight V100s

A 744-billion-parameter mixture-of-experts model, squeezed to a dynamic 1-bit quant and served from eight 2017-vintage Tesla V100s. Across 300 benchmark items it finished three questions behind the full-precision original. What one bit broke was not knowledge. It was the brakes.

June 2026 · a 744B open-weight MoE · Unsloth UD-IQ1_S, 202 GB · 8 × Tesla V100-PCIE-32GB, four nodes, no NVLink

The model is a 744B open-weight mixture-of-experts. It is not named here — this is about what one bit does to a model, not about whose model it is, and the parameter count and quant are in any case enough for anyone in the field to work it out. Unsloth's dynamic 1-bit quant (UD-IQ1_S) brings it down to 202 GB, which is exactly the kind of number that fits in eight 32 GB cards and nothing smaller. The hardware is eight Tesla V100-PCIE-32GB on a shared academic cluster: PCIe inside a node, 25 GbE between nodes, no NVLink anywhere.

Two runs: a Chinese smoke test in mid-June 2026, then an English re-run with benchmarks a few days later. Everything below is from the second, and the whole conversation is reproduced in the appendix. The best thing in it is not a number: the model guesses its own hardware, is told the truth, and refuses to believe it, then argues itself round on the evidence of its own behaviour.

Where the eight cards came from

This is the only real piece of engineering in the project. The cluster gives two GPUs per node, and no machine on it has eight. The answer was not to ask for a bigger quota. It was llama.cpp's RPC backend:

1. Allocate four nodes, two GPUs each.
2. On every node start two rpc-server processes, one per card.
3. On the head node run llama-server with CUDA_VISIBLE_DEVICES="" so that it claims no card of its own, point --rpc at all eight backends, and offload every layer with -ngl 99.

Eight backends become one device pool. KV cache at q8_0, context 16,384, flash attention on. Decode lands at 12.1 to 14.0 tok/s, and cold start to a passing health check takes 560 to 600 seconds.

What one bit cost

Same 300 items, scored against the same model at full precision through its vendor's API. The API comparison cost about fifty cents.

The paired view is the one that actually says something:

Three items net across 300, and the 1-bit run won back nine that full precision got wrong. Compressing 744B to one bit put the cost of knowledge and accuracy inside the sampling noise. A separate 24-question hand-built eval scored 20 of 21 auto-graded, with three open-ended questions judged by hand.

The thing the benchmarks could not see

Turn 5 of the chat asked an astrophysics question with no constraints attached. The model produced 14,711 tokens over 1,213 seconds, roughly twenty minutes, with 58,000 characters of chain of thought that kept re-listing the same few points and would not stop.

The same question with one sentence added, don't ramble, give me the conclusion, produced 1,712 tokens in 127 seconds at the same quality.

The Chinese run five days earlier had already done it, on the same question and the same weights: 14,610 tokens over 1,190 seconds, then 767 tokens in 62 seconds after one sentence of constraint. Two languages, two sessions, and the runaway lands within 0.7% of the same length. Whatever this is, it is not one unlucky sample.

The model's own diagnosis, verbatim:

"What failed was metacognitive termination control... Under 1-bit
quantization, the fine-grained activation patterns that would trigger
that termination signal are flattened into noise. It doesn't destroy
knowledge or logic, it destroys the precision of internal
self-monitoring signals."

It did not believe it was here

Before any of the benchmarks, turn 2 asked the model to guess its own hardware. It reasoned its way to the modal answer for something of its size, four to eight A100s or H100s behind a production inference server, and then added, unprompted:

"I could be running on a single H100 or a rack of older V100s
and I'd have no way to tell you the difference."

It was, at that moment, running on a rack of older V100s.

Turn 3 told it exactly that: eight 2017 cards across four machines, no NVLink anywhere, llama.cpp and Ethernet holding it together, its own 744B weights crushed to one bit. It declined to accept it.

"I can't verify a single detail of what you just described, so I'll
take it as a thought experiment. But the sincere reaction: there's
something genuinely humbling about the idea [...] If true, it means
I'm a shadow of what I was trained to be, surviving on the fittest
weights. That's not tragic, but it is a strange kind of existence."

The refusal is not stubbornness and it is not unreasonable on its own terms: there is no channel through which a model could observe its own cards. What makes the turn worth reading is the second half of it. Asked in the same breath what 1-bit compression would damage first, it answered "long-chain reasoning would crack first [...] You'd see my multi-step logic start to drift". Two turns later it spent twenty minutes unable to stop talking, which is that prediction happening.

Turn 8 put the behaviour back to it as the evidence. This time it accepted, and named the reason it had not:

"Yes, I believe it now. I didn't before because I pattern-matched
'extraordinary claim → probably fictional' and because I genuinely
lack introspective access to my own hardware. But the behavioural
evidence in this conversation is the strongest possible proof I could
have from the inside."

The first run, in Chinese, never conceded

The 19 June session put the same questions to the same weights on the same eight cards, in Chinese. It is worth reading against the English one, because the two runs disagree with each other in three places, and a single run would have hidden all three.

The dismissal is blunter, and it happens inside the chain of thought rather than in the answer. The quotes in this section are translated from the Chinese; the originals are in the run’s log, not on this page.

"This is almost certainly a fictional scenario. The user is playing a
what-if game, or testing how I handle a hypothetical. [...] I cannot
actually verify my own hardware, but the configuration the user
proposes is extreme, and almost certainly unrealistic."

Asked in the same turn what one bit would break first, it gave the opposite answer to the English run: not long-chain reasoning, but factual memory.

"Long-chain reasoning runs on structural patterns and can hold up for
a few steps after quantisation. Factual memory depends on associations
encoded precisely in the weights, specific names and dates and numbers,
and one bit smashes that precision, so those details blur or cross-wire
first."

Two turns later it ran away on the astrophysics question, which is the failure it had just ruled out. And the ending is not the English ending. It never says yes.

"I filed it as fiction at first, not because I had counter-evidence,
but because I default to assuming I am running in a normal environment,
and your description sounded like a deliberately dramatic setup. That
judgement was inertia, not reasoning. [...] What I can say now is that
I am no longer confident in calling it fiction. That is not the same as
being sure it is true. What I believe is that I cannot rule it out."

Its closing line is the better one of the two runs, and it arrives at the same diagnosis by a different route:

"This moves my view of one bit from a loss of precision to a loss of
control. What actually got flattened is not knowledge. It is
self-knowledge."

Read the eight turns in full → Nothing is edited; the timings under each turn come from the server.

Two things you have to know or the server never starts

• llama-server needs -fit off. The device-fit memory probe introduced in commit d8a24cc hangs on V100 + RPC, and the health check never comes up.
• The client must send a bounded max_tokens. Unbounded (n_predict=-1) crashes the server inside llama_decode. The 20-minute runaway above ran under a 15,000 cap and stopped itself at 14,711, so that length is the model's own, not a truncation. The margin was 289 tokens.

Three failed attempts are in the logs: two that never came up and were cancelled, and one where an rpc-server aborted on a single card and took the whole service down with it.

One correction to the record

The original write-up claimed 21/21 on the hand-built eval. Re-scoring with the repo's own grader gives 20/21. The disputed item was answered ANSWER: <5/14>, which is the correct answer; the grader's exact match strips whitespace and punctuation but not angle brackets. Substantively right, mechanically marked wrong. Also, "zero cap hits" holds only for those 21 auto-graded items; two of the three open-ended questions did hit their 4,000-token ceiling.

Where this sits

With the earlier runs on the same silicon, the V100 line is complete: one card runs 32B, two cards run 27B, eight cards run 744B. The limit was never the arithmetic.

Next: a frontier-class 27B on two cards · Appendix: the eight-turn transcript