An agent pays for its context once, then keeps paying
Across 98 real coding-agent sessions, the model reads 386 tokens for every one it writes. Almost all of that reading is the same context, again, every turn. Whether the prefix cache holds is worth 8.1× on the bill, which is a larger swing than the choice between one model tier and the next.
Most cost intuitions about language models were formed on a chat turn: a person types a paragraph, the model writes a paragraph back, and the two sides of the bill are roughly the same order of magnitude. An agent session does not look like that at all, and the difference is not a detail.
The shape of an agent session
In 98 real coding-agent sessions from a public trace dataset, the median input length per request sits between 109,000 and 181,000 tokens, and the median output is 212 to 217 tokens. The model reads a small book and writes a paragraph. Aggregated over the sessions, prefill to decode comes out at 386 : 1.
That ratio is not a sign of a badly written agent. It is what a tool-using loop is: every turn re-sends the task, the files already read, the transcript so far, and the tool definitions, so that the model can decide one more thing. The context is not consumed. It is resident, and it is paid for again on every turn it stays resident.
Hashing every request in 64-token blocks gives the exact figure for how much of that reading is repetition. The best hit rate any cache could achieve on these traces is 0.9898: 99 percent of everything the model reads, it has already read. Put the other way round, each distinct token is read about 98.3 times before the session ends.
What the cache is worth
If 99 percent of the input is repetition, then whether the serving stack recognises the repetition is a first-order cost variable. Priced at public list rates, with the cache working as well as it possibly can against the cache missing entirely:
| Price tier (in / out per M) | Cache working | Cache missing | Ratio |
|---|---|---|---|
| $5 / $25 | $2,052 | $16,679 | 8.1× |
| $3 / $15 | $1,231 | $10,008 | 8.1× |
| $1 / $5 | $410 | $3,336 | 8.1× |
Totals across the 98 sessions. The top tier works out to $20.94 per session with the cache and $170.20 without.
The ratio is identical down the column, which is the point worth carrying away. Moving down a model tier divides the bill by roughly five. Losing the prefix cache multiplies it by eight. The serving decision is bigger than the model decision, and it is invisible in every price comparison that quotes dollars per million tokens.
Why this is a cost note and not a caching note
Because the same arithmetic decides whether a self-hosted deployment makes sense at all. A hosted API bills you for the repetition unless its cache catches it; a machine you own has already been paid for and the repetition costs you only time. That asymmetry is exactly where local deployment stops being a hobby and starts being a budget decision, and it is why the honest comparison is never "dollars per million tokens, hosted versus self-hosted".
It also means an agent workload stresses a server in a completely different place than a chat workload. Chat is decode-bound and latency-shaped. Agents are prefill-bound and cache-shaped. A deployment sized on chat traffic will be sized wrong for agents by a factor with three digits in it.
The two numbers nobody publishes
I went looking for public data to close the local-versus-hosted comparison properly, and two things are missing everywhere:
Machine-time depreciation. The standard public benchmarks explicitly forbid reporting price, so the only hourly figures available are vendor list prices for rented hardware, which is not what an owned machine costs.
Duty cycle. Every public benchmark runs the machine flat out. Nobody reports how many tokens a real deployment actually produced over a year. For a small team's own hardware this is the dominant term, and it is simply absent from the literature.
If you run a deployment and have either of those numbers, they are worth more than another throughput benchmark. I would like to hear from you.
What this is and is not
This is an arithmetic reconstruction: real traces that somebody else collected, re-priced at published list rates. It is not a measurement of my own hardware, and the dollar totals move whenever a vendor changes a price. What does not move is the structure, because the 386 : 1 ratio and the 0.9898 hit rate come from the traces themselves.
The sessions are coding agents. I would expect the ratio to be different, probably lower, for agents that work over documents or spreadsheets, and I have not found a public trace set for those. That gap is the next thing on this line of work.