A 33B BF16 video model on FP16-only silicon

The obvious plan is one line long: cast to fp16, split the model across the two cards. It fails for six independent reasons. Two are fp16 dynamic range, one is memory, two are kernel availability, one is a misconfigured offload, and one of the six we inflicted on ourselves while fixing another. None of them is about arithmetic speed, which is the thing Volta is usually assumed to be bad at.

The machine, measured rather than quoted

Two consequences shape everything below. "Just run it as released" is not an answer: running the 50-block stack in the checkpoint's own dtype throws away an order of magnitude of arithmetic. So the constraint for the whole project is keep the stack in fp16, and the work becomes entirely about making fp16's dynamic range survive a model that was never trained inside it.

And quantisation is not a speed tool here. Volta's integer path is slower than its fp16 tensor cores, so on this hardware INT8 costs quality and buys no throughput. It is a memory tool, and memory turned out not to be the binding constraint after one particular observation.

Six blockers

The residual stream was never representable

With the accumulator widened to fp32, the true values become visible for the first time: 97,307 at block 0, already 1.49× over the fp16 ceiling, 325,926 by block 10, and a trajectory peak of 7,480,017, which is 114× the fp16 maximum. Everything measured before that was fp16 clipping on the way out.

The load-bearing reading is the other one: the tensors entering the matmuls are nowhere near the ceiling (single digits, against 65,504). What is large is the accumulator. Widening only the accumulator therefore costs nothing on the tensor cores, and the GEMMs stay in fp16 at full rate.

Why the overflow fixes are free

Inside a transformer block, every linear layer is bias-free. That is verified against the checkpoint index, not assumed, and it is the single structural fact that does the most work in this project. A bias-free map is exactly homogeneous, so f(x/32)·32 == f(x), and 32 is a power of two, so dividing an fp16 tensor by it only decrements exponents. Not one mantissa bit is lost. The compensating multiply lands in the fp32 accumulate that already exists.

The SwiGLU projection cannot use the same trick, because the activation is not homogeneous. But the overflow is in the elementwise product, not in the GEMM, so widening only the gating product to fp32 fixes it while both GEMMs stay on the tensor cores. Measured: the projection output peaks at 0.008× the ceiling, with 130× to spare, while the gated product reaches 110,154 and crosses the ceiling at step 4. The fp32 gating is not insurance; without it, every step from the fourth onward overflows.

At 19,300 rows a fixed scale is the wrong shape of fix: a single constant must be large enough for the worst row, which pushes every other row toward subnormals for nothing, and it still fails the moment one row is more extreme than the constant anticipated. Scaling per row by a power of two stays exact (bias-free implies row-wise homogeneous) and, because the compensation undoes it exactly, it reports true magnitudes: the largest was 2.51e6, above what a fixed 1/32 can represent. That row genuinely had run out of range.

One of the four was not an overflow at all

One site produced a single inf while its finite maximum sat at 6,172, more than ten times below the ceiling, with no neighbour anywhere close. The cause was allow_fp16_reduced_precision_reduction, which defaults to true and lets cuBLAS accumulate the split-K partial sums of an fp16 GEMM in fp16. A 14,336-deep contraction can therefore blow past 65,504 in a partial sum long before the final dot product, worth a few thousand, is ever formed. Turning it off produced the first complete 20-step trajectory with picture and sound.

This explanation is held to a lower grade than the rest: consistent, not established. Two flags changed rather than one, the runs diverge from the first block so it was a global reroll of every fp16 GEMM, and what it removed was a single marginal element. Reroll and mechanism predict the observation equally well.

39.3% of the model is a lookup table

The adaptive-normalisation projection is 13.006B parameters, 24.2 GiB, 39.3% of the model, and its input is only the timestep embedding. It does not depend on tokens at all. For a 20-step run the entire per-block projection is about 0.5 GFLOP. Thirty-nine percent of the memory is doing a rounding error's worth of the compute.

This also explains the vendor's own compact release, which is about 21B parameters: the same 13B removed. Ours comes out in fp16, the only precision with tensor cores on this hardware, rather than INT8, which buys nothing here.

Two details that cost time. The two modalities walk different noise schedules, zipped per step, so an N-step run visits 1 + 2(N−1) distinct timesteps, not N; the first table was sized for one schedule. And dropping 13B parameters moved the automatic device split, which put the output heads on the wrong card. They are pinned explicitly now, with an assertion, because anything that changes the parameter count can move that boundary again.

The picture was wrong, and it was not our arithmetic

For most of the project the video carried saturated colour blocks that survived every numerics fix. Three measurements settled it, in order.

The corruption is quantised to the transformer's own token grid. Chroma edge strength on the 32-pixel grid is 2.40 horizontally and 3.36 vertically, against 1.19 and 1.29 for boundaries that are on 16 but not 32. And 32 pixels is exactly one video token. VAE tiling is ruled out twice over: vertically the frame is smaller than one tile, so no vertical seam can exist, yet the vertical edges are the strongest in the frame.

The VAE is cleared independently: one latent decoded three ways gives a maximum fp16 deviation of 0.069 against a signal range of 3.83, with zero non-finite outputs across 437 leaf modules.

And the entire recipe is cleared by a control that should have existed on day one. Running bf16 with the stock block forward and zero patches, same seed and prompt and canvas, produced 6.4% fewer corrupted blocks: the same fox in the same pose behind the same cyan blocks in the same places.

What was left was the canvas. The small canvas is legal (the dimensions divide correctly) but not supported: it gives 60 video tokens per latent frame, and the artefacts are quantised to exactly that grid, roving and sporadic, surviving a complete change of arithmetic. That is what a model asked for a canvas it was never trained on looks like.

What this does not establish

There is no bf16 reference at the working canvas, and there cannot be one on this hardware. bf16 has no memory-efficient attention kernel on sm_70, so attention falls back to the materialising backend, which at 19,300 rows needs a 75 GiB allocation. This is the concrete counterexample to "anything an H100 can run, a V100 can run more slowly": not slower, unavailable.

So the honest statement is not "fp16 is accurate here". It is: fp16 moves the latent substantially (29.4% of range against the stock bf16 path), and at the only canvas where the comparison can be run, the output is already degraded enough that the change does not show. Whether the recipe is acceptable at a canvas that works is unmeasured, and unmeasurable on this machine.

Phase two: making it fast

Before optimising anything, every region of every block was timed with CUDA events on the production forward.

Attention is 60% of the step and runs at 30 TFLOPS against the 91.8 the same silicon does on a dense matmul. The PCIe handoff between cards costs 0.18 s, so the pipeline split was never communication-bound: it was bound by only one card computing at a time. (A third finding: the diagnostic forward used for every previous measurement costs 1.03× by itself, so every number before this phase was 3% pessimistic.)

Tensor parallelism, 1.65×

Four facts make sharding unusually clean on this model:

• Rotary embeddings broadcast over the head axis, so splitting heads 28/28 needs no communication inside attention at all.
• Every linear is bias-free, so row-parallel partial sums add without a correction term and the fp16 recipe survives intact.
• The attention mask is empty on this path, so there is nothing to shard or replicate.
• The sequence is packed and contiguous: no cross-attention, no ragged batching.

Two all-reduces per block, 100 per step, measured at 27.75 ms each — 2.78 s/step of communication that the sharded compute more than pays for.

On every quality instrument tensor parallelism is indistinguishable from the pipeline split: the shot cut lands on the same frame, marginally sharper, 1.1% less motion, audio spectral cosine 0.997.

Sparse attention is dead here, and the measurement is the point

97.8% of rows are video. The proposal was to let each video query attend to its own frame plus all non-video keys: 940 of 19,300 keys, benchmarked in isolation at 10.8× faster, one SDPA call, no kernel to write. The pass/fail rule was committed before the run. Over 1,400 sampled (step, block, head) combinations:

top-940 mass       (can ANY 940-key scheme work?)   median 0.890
spatial+nonvideo   (does THIS scheme work?)         median 0.103
head classes: {'DENSE': 56}    0 of 56 heads clear a 0.7 bar

The sparsity is real; the locality assumption is false. Depth does not rescue it, the schedule does not rescue it, and the best single head reaches 0.374. Finding the right 940 keys means ranking all 19,300, which is the full attention score computation that sparsity exists to avoid. That is the entire difference between static and dynamic sparsity, and a projected 3× on this route is retracted.

Step caching: the speed is arithmetic, the quality is a different sample

A skipped call costs 0.15 s against 19.48 s computed, so the speedup is exactly 1/(1−skip) and four arms confirm it to within 1%. The interesting result is that the published indicators are worse than no indicator at all:

The 41% arm skips fewer steps, runs slower, and loses on the cut, the smear, all three audio numbers and both pixel distances. The skip patterns explain it: the indicator skips two calls in a row very early, while the sample's global structure is still being decided, then computes every one of the last ten, when nothing is left to decide. "Never skip the opening steps" is not a heuristic, it is the whole result, and a uniform pattern satisfies it by construction with nothing to calibrate.

Both published indicators are essentially uncorrelated with what they predict (Pearson −0.160 and −0.015). Fitting the standard degree-4 polynomial gives an in-sample R² of 0.791 and a leave-one-out R² of −104.6, which is worse than predicting the mean. And the quantity actually reused never changes by less than 11% anywhere in the schedule, so there is no quiet region to exploit: every skip is a real substitution, not an approximation of one.

The sting: the loop was 22% of the run

After all of that, the setup stages of a 50-step job were timed for the first time.

Reading the checkpoint takes eleven seconds. Wrapping it takes thirty-eight minutes, to enable eleven seconds of work, on a model never touched again. The 1.65× on the loop is about 12% end to end.

And 37.8 minutes is not a constant: three timings of the same line gave 2267 s, 775 s and 298 s, a 7.6× spread. The mechanism is visible in the two numbers next to each other. Loading returns in eleven seconds because the checkpoint is memory-mapped and nothing has been read yet; the reading happens inside the offload wrapper when it touches every leaf module. That line is not hook overhead, it is 63 GB coming off a shared filesystem at 28, 83 and 211 MB/s on three different days.

Both large items are constants — the table depends only on the schedule, the prompt state only on the prompt — so both go to disk. Cold start went from ~25 minutes to under two, and the cached prompt state that replaces 775 s of setup is 5.5 MB. Four independent productions of the same clip share one SHA-256, so the caching changed nothing at all, and the checksum discriminates: a differently-parallelised run hashes differently.

One step further: the text encoder does not need a GPU. The offload machinery exists to walk a 64 GB model through a 32 GB card; on a CPU node with 187 GB of RAM it is not made faster, it is absent. Running it there in bf16 beats fp32 by a wide margin while computing 6.8× slower, because fp32 has to read and upconvert twice the bytes and the whole route is read-bound.

Two failure modes explain almost every retraction

This project retracted a long list of claims. Recording them is not penance: two failure modes account for nearly all of them, and both are cheap to defend against once named.

A sample reported as coverage. Five times, including once during an audit that was correcting earlier instances, and once after the failure mode had been written up as a lesson twice. "4.7× margin across all 50 blocks" came from a table that sampled every fifth block and never printed block 49 — and block 49 is where the next failure was. "Never exceeds 2,474" came from a run that went non-finite at block 13, so blocks 14 to 49 had no numbers at all; the true value is 13,792.

An acceptance bar set as though the system does not amplify. Three times, in a system whose defining property is amplification. The general form: a bit-exactness criterion on the output of a chaotic iteration is not a correctness criterion. It tests determinism, which is a different property, and it fails for correct code. Any bar downstream of a 50-block stack has to be set against a perturbation already accepted, not against zero.

And one instrumentation defect worth fearing more than the others: folding a scale factor into the weights at setup time. Under CPU spill, the framework restores offloaded weights from its own copy at every forward, so the fold never takes on them while the compensating multiply in the forward applies everywhere. It would have amplified part of the stack 32× while raising nothing, and produced plausible video from a wrong model. It was caught by reading, not by a test. The same trap has a reading form: a weight is only valid inside its own module's forward; read it afterwards and you get plausible garbage.

A structural finding that fell out of the sweeps

Reading the modulation weights across all 50 blocks, three modalities and six slices, the complete list of exactly zero slices in the entire model is one line: four slices in the final block's text path. Those are precisely the parameters with no gradient path to the loss — the output heads discard text rows after the stack, so anything that only moves the text residual after attention in the last block can never reach the loss, while the two slices that shape the text keys and values other modalities attend to stay live. Four predicted dead, four measured dead, zero false positives across 900 slices.

Two numerical consequences: a zero gate does not neutralise an overflow, because in fp16 zero times infinity is NaN; and exactly-zero rows break any adaptive per-row scheme, which is why the per-row scaling carries a clamp.

Previous: a frontier-class 27B on two cards · Next: the same hardware, two orders of magnitude cheaper per clip