Cost notes

What running a model actually costs

Three field notes on the cost side of AI adoption. Not what a model can do, and not what a token is rented for, but what it takes to get a real task finished on real hardware, and where that number is decided. Short version: it is decided by utilisation, by repetition, and almost never by the price on the vendor's page.

July – September 2026 · measured on local hardware, on real agent traces, and on twenty months of archived list prices

A model that is free to download is not free to run, and a model that is cheap per token is not cheap per task. Most of the interesting distance between those sentences is measurable. These are the measurements, written up as I go.

The notes

Note 01

Throughput is not the number that sets cost

One 27B model on two 2017 cards. Allowing eight requests in flight instead of one multiplies output by 5.3× on an identical workload, and the hardware bill does not move. Cost per token has utilisation in the denominator, and utilisation is a scheduling decision.

27.55 → 146.68 tok/s, same 60 prompts · TTFT 0.23 s → 3.53 s across the sweep

Note 02

An agent pays for its context once, then keeps paying

Across 98 real agent sessions the model reads 386 tokens for every one it writes, and 99 percent of that reading is repetition. Whether the prefix cache holds is worth 8.1× on the bill, a bigger swing than moving down a model tier.

386 : 1 prefill to decode · best possible hit rate 0.9898 · $2,052 vs $16,679

Note 03

Per-token price is rent, not cost

The cheapest model per token costs fourteen times the cheapest model per task, because it spends nine thousand tokens thinking about each question. And while the rental rate sat still for five months, the real cost of the job fell six-fold.

14.12× selection error · rent flat at $0.06 while cost per task fell 6.25×

The thread through them

The three notes are the same argument at three scales. At the level of a purchase, the card is not the cost; the fraction of its life spent serving is. At the level of a session, the prompt is not the cost; the number of times it is read back is. At the level of a price list, the token is not the cost; the task is.

Each of those gaps is an order of magnitude or close to it, and none of them is visible in the number people usually quote. That is the working hypothesis behind this whole line of work: the reason model adoption is slower than the headline price curve suggests is not that models are expensive, but that the published price is measuring something else.

What is still missing

Two numbers would close the local-versus-hosted comparison properly, and neither exists publicly. One is machine-time depreciation: what an hour of an owned machine costs, as opposed to what an hour of a rented one is listed at. The other is duty cycle, meaning how many tokens a real deployment actually produced over a year rather than how many it produces flat out on a benchmark. Every public benchmark runs the hardware saturated, so the second term is missing everywhere, and for a small team's own hardware it is the dominant one.

If you operate a deployment and have either figure, even roughly, I would like to hear from you. It is the shortest path from these notes to something that can actually answer "should we host this ourselves".

Related: the price frontier of intelligence, twenty months of the public leaderboard on one ruler · serving modern models on 2017 silicon, four experiments on what old hardware still does.