A frontier-class 27B, on cards from 2017
Qwen3.8-27B trades punches with a frontier hosted model on agentic coding, ships under Apache 2.0, and is a 16.5 GB file at 4-bit. That combination is the story: the interesting number here is not tokens per second, it is where the weights are allowed to live.
This one is not a stunt. It is a residency: the model stays up, a watchdog keeps it up, and a machine at home talks to it over a tunnel. The question was whether a 27B model with a hybrid linear attention stack would even run on sm_70, and then whether it would be pleasant.
Sampling uses the thinking-profile defaults embedded in the GGUF (temp 1.0, top-p 0.95, top-k 20) with max_tokens set high enough to be a backstop rather than a truncation. Truncated samples are excluded from the medians. Capping output for a benchmark does not make it cheaper, it makes the model think less, which is a different measurement.
Why this model matters
Qwen3.8-27B shipped on 14 August 2026: 27.78 B parameters, Apache 2.0, a native 262,144-token context extensible to a million, text and images and video, and 64 layers of mixed Gated DeltaNet and gated attention (model card). On its vendor's own benchmark card it is put head to head with Claude Opus 4.6 Max — and on agentic coding it wins more of those columns than it loses.
Data
| benchmark | Qwen3.8-27B | Opus 4.6 Max | difference |
|---|---|---|---|
| IFBench | 79.5 | 62.5 | +17.0 |
| SWE-bench Pro | 61.7 | 53.4 | +8.3 |
| CoWorkBench | 70.7 | 68.2 | +2.5 |
| LiveCodeBench v6 | 90.3 | 88.8 | +1.5 |
| GPQA Diamond | 89.2 | 91.3 | -2.1 |
| Terminal-Bench 2.1 | 73.0 | 78.2 | -5.2 |
| NL2Repo-Bench | 42.3 | 47.6 | -5.3 |
| Humanity's Last Exam | 30.8 | 40.0 | -9.2 |
Read that chart with its provenance attached. Those are the vendor's own numbers, published alongside the weights. The "matches Opus 4.6 on coding" line that went round the coverage (VentureBeat) traces back to this same card rather than to an independent evaluation. The vision columns on it favour Qwen by margins running past 20 and even 70 points, which says more about how that comparison was set up than about either model, so they are left out above.
The independent number that does exist points the same way. Artificial Analysis puts it at 52 on their Intelligence Index, a composite of nine evaluations, and ranks it first of 135 open-weight models in its size class. Their listed serving price for it is $0.43 and $3.10 per million input and output tokens, against $5 and $25 for Opus 4.6.
The claim worth making is not "a 27B beats a frontier model" — benchmark cards are marketing surface and this one is no exception. It is narrower and harder to argue with: the class of behaviour that used to require somebody else's datacentre now fits in a 16.5 GB file you can download. Everything below is about where that file can run.
One card, or two, and what the second one buys
The quantisation ladder is the part that matters, and it is published: 16.5 GB at Q4_K_M, 22 GB at Q6_K, 29 GB at Q8_0, against 54.7 GB for the released BF16 weights (GGUF builds).
- One consumer card is enough. At Q4_K_M the whole model is 16.5 GB, which leaves room for a working context on a single 24 GB card.
- One V100 is enough, on paper. Q6_K is 22 GB of the 31.73 GiB a card here actually offers, and at the KV cost measured further down — 64 KiB per token — the remaining ~9 GiB is on the order of 140K tokens of context. That is arithmetic from two measured numbers, not a run: I have not served it from a single card.
- What the second card buys is capacity, not speed. The pair is what makes Q8_0 (29 GB, near-lossless) plus the full 262,144-token context plus the 3.2 GB draft head fit at once — 48.9 of 64 GiB. Under layer splitting a single stream runs at about the speed of one card, which was already the finding on this hardware the first time it was measured.
And that is a consequence of the interconnect, not of the model. These cards have no NVLink, so layer splitting is the only sensible split: card one runs the first half of the stack, card two the second, and only one of them is working at any instant. With NVLink you would shard each layer across both cards instead, putting both cards' memory bandwidth behind the same token — on a decode this cleanly bandwidth-bound that should land somewhere around 50 to 60 tok/s. That is an expectation from the bandwidth arithmetic, not a measurement; there is no NVLink here to test it on.
Baseline against speculative decoding
Data
| task | baseline tok/s | MTP tok/s | gain | accepted |
|---|---|---|---|---|
| prose | 23.58 | 35.14 | 1.49× | 34.6% |
| technical, Chinese | 23.58 | 36.12 | 1.53× | 36.4% |
| code | 23.38 | 45.72 | 1.96× | 57.1% |
The baseline varies by less than 1% across three very different tasks (23.38 to 23.58), which is the signature of a cleanly bandwidth-bound decode: 23.58 tok/s × 28.6 GB = 674 GB/s against 900 GB/s of HBM2, 75% bandwidth efficiency.
Speculation costs about 10% of prefill and returns 53% of decode. Acceptance on code is far higher than on prose (57.1% against 34.6%), and the gain follows it.
Against a modern low-bandwidth box
| GB10 desktop | 2 × V100 | |
|---|---|---|
| baseline | 7.87 | 23.58 |
| MTP, draft depth 3 | 18.38 | 36.12 |
| speculative gain | 2.32× | 1.53× |
| memory bandwidth | 273 GB/s | 900 GB/s |
| bandwidth efficiency | 82% | 75% |
The nine-year-old cards are three times faster in absolute terms, tracking the 3.3× bandwidth ratio almost linearly. The speculative gain is lower here, and that is where it would be easy to write something wrong: acceptance in this round fell between 34.6% and 57.1%, and in the other round between 47.2% and 66.7%. Both are single runs at temperature 1.0, and acceptance at a fixed setting is known to swing thirty points on its own. That difference is probably noise and should not be reported as a hardware result.
Memory, and why the full context is nearly free
Data
| configuration | GPU 0 | GPU 1 | total | cold load |
|---|---|---|---|---|
| context 32,768 + MTP | 17,396 MiB | 14,726 MiB | 31.4 GiB | 11 s |
| context 262,144 + MTP | 27,252 MiB | 22,790 MiB | 48.9 GiB | 30 s |
Memory before and after a generation differs by 4 MiB, so the KV cache and the compute graph are allocated once at startup: a long context will not creep into an OOM halfway through a session.
Measured KV cost matches 64 KiB per token, and the reason is architectural. Of 64 layers only 16 are full attention (one in four); the rest are Gated DeltaNet, whose state does not grow with sequence length. A 27B model gets a 256K context on two V100s because of how it is built, not because of what it was quantised to.
Vision costs almost nothing
Attaching the multimodal projector at the production context adds 0.83 GiB, taking the pair to 49.7 of 64 GiB. The vision tower on sm_70 was an open question before the run and is now closed.
llama.cpp warns that Qwen-VL models want at least 1,024 image tokens. That warning is about grounding, meaning coordinates and boxes. On a 900×560 test image with three shapes and three lines of text, the default 584-token path and the 1,024-token path returned word-for-word identical answers. Default is enough; pay the extra tokens only if you actually need grounding.
The first version of that test image used PIL's default bitmap font at 11 px, and the model read a capital O as a zero. Switching to a 46 px vector font made both configurations perfect. That was a defect in the test image, not in the model, and it is written down here so that nobody later mistakes it for evidence.
Keeping it up: three separate problems
Keep-alive is a cron job that is also the service
Slurm ships scrontab, and its manual says the next job in a series is not submitted until the previous one has completed. That single sentence is a keep-alive: schedule the launcher every ten minutes against a four-day wall time, and a healthy instance simply causes every subsequent tick to be skipped, while a dead one is replaced within ten minutes on whatever node is free.
The design point that matters: the long-lived instance must be the cron job. Submit a separate batch job as the service and every tick still fires, hits the single-instance guard, and exits: 576 pointless launches and 576 log files over four days. Let the cron job hold the service and the scheduler skips the ticks for free.
Two traps, both documented and both expensive to rediscover: cancelling the job also cancels the cron entry and comments the line out, so restarts have to go through a wrapper rather than a bare cancel; and scrontab ignores the user environment, so every path in the command must be absolute.
The tunnel can only be dialled one way
The elegant design is a reverse tunnel from the compute node to a fixed address. It is not available: compute nodes are refused by the login nodes' SSH policy (same key, same agent, works between compute nodes, refused to login nodes, and a different key type changes nothing), and compute nodes have no route to the outside world at all. So the connection is dialled inward from home. There is no choice to make, which is worth knowing before spending an evening on it.
Watching the process is not watching the service
A forwarded port resolves its target at connection time on the hop in the middle. When the service moves to a different node, the SSH process stays perfectly healthy and every new connection fails. Monitoring the tunnel process can never detect that the service has moved house.
So the watchdog does two things instead: it actually calls the health endpoint every 20 seconds and rebuilds only after two consecutive failures, and it re-reads the current node name every 60 seconds and rebuilds immediately if it changed. End-to-end recovery is one cron tick plus 30 seconds of model load plus up to a minute of discovery. The bottleneck is the scheduler, not the tunnel, which is why minute-granularity is the right amount of engineering here.
Two build traps on sm_70
- Pass the CUDA toolkit root explicitly. Left to itself, CMake resolves the CUDA runtime to a node-local real path while the built CUDA library records the symlinked one, and the final link fails with a screen of
undefined reference to cudaMalloc@libcudart.so.12. - The web UI assets are gzip-compressed into the binary. Fetching the root URL with a plain client returns
415 Error: gzip is not supported by this browser. Adding--compressedfixes it; browsers were never affected.
A client-side trap worth its own paragraph
Wiring the server into an agent client, the switch that enables image input was modalities, not attachment. The client accepted attachment: true and displayed it in its own config dump, but internally checked a different field. With it unset, attaching an image did not raise an error: the file was silently intercepted client-side and the model received a sentence saying it does not support image input. It looks exactly like a model limitation, and the image never left the machine.
The general technique, worth keeping: start the client in server mode and query its own config endpoint to see what the client believes about the model. It is much faster than guessing at config keys. And a key file must not end in a newline, or the newline goes into the Authorization header verbatim.
What this settles
- Hybrid linear attention runs fine on
sm_70. That was the largest unknown going in. - Turn the draft head on. 1.53× median, near 2× on code, for 3 GB and 10% of prefill.
- Q8_0 is the comfortable setting here: 36 tok/s, a 256K context, near-lossless weights. There is no reason to drop to 4-bit for speed on this hardware.
Ask about these notes
A small agent with three tools — search, read, outline — over the 44 sections of these notes. It looks things up before answering and links the section it read. It will tell you when the notes do not cover something rather than guess.
Grounded only in these pages, and it cites the section it read. A question costs your browser about a third of a second of arithmetic — that is the spam gate, and it stores nothing.