Throughput is not the number that sets cost
A GPU-hour costs the same whether you push 27 tokens per second through it or 436. On two 2017 Tesla V100s serving one 27B model, going from one concurrent request to eight multiplies output by 5.3× on an identical workload, and the card bill does not move. Which means the denominator of cost per token is utilisation, and utilisation is a scheduling decision, not a procurement one.
The question behind this note is a budgeting question, not an engineering one. If you are asked what it costs to serve a model, the honest answer has a denominator in it, and the denominator is not the card. It is how much of the card's time is spent producing tokens that somebody asked for.
So: one model, one quantisation, one pair of cards, and the only thing changed between runs is how many requests are allowed in flight at once.
The sweep
Qwen3.6-27B at AWQ 4-bit, tensor-parallel over two Tesla V100-PCIE-32GB, served by TurboMind, driven with ShareGPT conversations.
| In flight | Output tok/s | TTFT | Time per output token | Prompts in the run |
|---|---|---|---|---|
| 1 | 27.55 | 0.229 s | 35.3 ms | 60 |
| 8 | 146.68 | 0.385 s | 42.8 ms | 60 |
| 16 | 145.98 | 0.802 s | 91.9 ms | 64 |
| 32 | 203.39 | 1.538 s | 127.4 ms | 128 |
| 64 | 324.49 | 2.201 s | 186.1 ms | 256 |
| 128 | 435.62 | 3.535 s | 286.1 ms | 512 |
The one clean comparison
Read that table with the last column in mind. Only two rows are a controlled comparison: concurrency 1 and concurrency 8 ran the same 60 prompts, 15,356 input tokens and 13,161 output tokens in both cases. Same work, same tokens, twice. Wall clock went from 477.7 seconds to 89.7 seconds, and output throughput from 27.55 to 146.68 tokens per second. That is 5.32×, and it is the number I would defend.
The rows at 16 and above scaled the prompt count with the concurrency, so they are different workloads. The jump from 27.55 to 435.62 is real arithmetic on real runs, but it is not a controlled A/B and I would not put "16×" in a slide without that sentence next to it. Note in particular that 8 and 16 are within a percent of each other on different prompt sets, which is exactly the kind of coincidence a cross-sample comparison produces.
What concurrency costs
Throughput is bought with waiting. Time to first token goes from 0.23 seconds to 3.53 seconds across the sweep, and time per output token from 35 ms to 286 ms. At 128 in flight the server is producing sixteen times the tokens and every individual user is reading them eight times slower.
This is the real shape of the decision, and it is why "what does it cost to host a model" cannot be answered without also answering "how slow are you willing to let it feel". A deployment tuned for one interactive user at a time is buying latency with money. A deployment tuned for throughput is selling latency for money. Both are defensible; only one of them is usually written down.
The cost arithmetic
Cost per token is the hourly price of the machine divided by tokens produced in that hour, and tokens produced in that hour is throughput multiplied by the fraction of the hour the machine was actually serving. Two terms, and only one of them is on a purchase order.
On this hardware the throughput term alone spans 16× between one request in flight and 128. The utilisation term, in a real campus or lab deployment, spans considerably more than that: a machine serving a bursty human population is idle most of the night. Multiply the two and the same card, running the same weights, can differ in cost per token by more than an order of magnitude with no hardware change at all.
Which is the practical conclusion: if you are trying to reduce the cost of serving a model, the first place to look is not the card, the quantisation, or the backend. It is the duty cycle. And the duty cycle is the one number almost nobody publishes.
What this does not show
One model, one quantisation, one backend, two nine-year-old cards. The absolute numbers do not transfer to an A100 or an H100 and are not meant to; what transfers is the shape, which is that the throughput term is large and free.
There is also no dollar figure anywhere in this note. The machine time came from a shared academic cluster, and I do not have an hourly rate for it that I would be willing to defend in public. Attaching a number I cannot source would undo the point of the note.
The benchmark was run in July 2026 and the project has not moved since. If you want the raw CSVs, ask.