Local Host Lab · Tools

Will it fit, and how fast?

A back-of-the-envelope calculator for running an open-weight model on one machine: memory for weights and KV cache, an upper bound on single-stream decode speed from memory bandwidth, and what a million tokens costs in electricity and amortised hardware. The formulas are on the page, and the estimate is checked against six measured setups from this notebook.

Estimates, not measurements · formulas below · runs in the browser, nothing is sent anywhere

Part of the Local Host Lab notebook. See also the hardware ladder, the cost-performance table and the sources.

Choose a model and a machine, or type your own numbers. Everything is recomputed as you type. The presets carry architecture values copied from each model's public configuration file and hardware values from vendor specification pages; both are listed, with dates, on the sources page.

Model

Preset Total parameters (billions) Active parameters per token (billions) Same as total for a dense model Weight format Bits per weight Effective, including scales Weight file size, GB (optional) Overrides the line above; most accurate

Context and KV cache

Context length, tokens KV cache precision Layers with a full KV cache Exclude linear-attention and sliding-window layers KV values per token per layer 2 × KV heads × head dim; MLA: rank + rope dim Sliding-window layers Window, tokens Runtime overhead, GiB Compute buffers and context; 1.5 is a rough default

Hardware

Preset Memory per device, GiB Usable fraction Unified-memory machines keep some for the OS Number of devices How the model is split Bandwidth per device, GB/s Realistic efficiency 0.75 to 0.82 measured for dense models below

Cost

Hardware price, USD Power while decoding, W Board power is an upper bound Electricity, USD per kWh Amortisation, years Hours of decoding per day Hosted price to compare, USD per million output tokens

Result

weights KV cache overhead usable memory

The formulas

Memory, in bytes:

weights   = total_params × bits_per_weight / 8        (or the file size)
kv_cache  = context × full_kv_layers × kv_values × kv_bytes
          + min(context, window) × window_layers × kv_values × kv_bytes
needed    = weights + kv_cache + overhead
fits      = needed ≤ memory_per_device × devices × usable_fraction

Here kv_values is 2 × KV heads × head dimension for grouped-query attention (one key and one value vector per KV head), and kv_lora_rank + qk_rope_head_dim for multi-head latent attention, which stores one compressed vector per token. A 16-bit cache uses 2 bytes per value, q8_0 uses 34/32, q4_0 uses 18/32. Linear-attention layers (Gated DeltaNet, Mamba) keep a fixed-size state and are left out.

Single-stream decode, in tokens per second:

bytes_per_token = weights × (active_params / total_params) + kv_cache
bandwidth       = per_device_bandwidth × (devices if tensor parallel, else 1)
upper_bound     = bandwidth / bytes_per_token
estimate        = upper_bound × efficiency

Generating one token means reading every weight that token uses once, and the whole KV cache built so far. On one stream nothing else is close to that cost, so memory bandwidth sets a ceiling. The ceiling ignores compute, kernel launch overhead, communication between devices, and the attention arithmetic itself, which is why the efficiency factor exists. It does not model speculative decoding (MTP or draft models), which in this notebook added 1.5 to 2.3 times on top, or batching several requests, which raises total throughput but not single-stream speed.

Cost per million generated tokens:

kWh per million     = watts / 1000 × (1e6 / estimate) / 3600
electricity         = kWh per million × price_per_kWh
hardware            = price / (estimate × 3600 × hours_per_day × 365 × years) × 1e6
total               = electricity + hardware

The hardware term assumes the machine does nothing else and is worth nothing at the end, and it charges every hour of the stated daily use to a single stream. It leaves out the rest of the computer, cooling, and the time of whoever runs it. Prefill (reading the prompt) is compute-bound, much faster per token, and not modelled here. Sizes are in GB of 10⁹ bytes, except device memory, which is entered in GiB as vendors label it.

How far off is the estimate?

The same arithmetic, applied to setups measured in this notebook, with the speculative decoding switched off so the comparison is fair. The estimate column is the bandwidth ceiling, before any efficiency factor.

Measured setup	Bytes / token	Ceiling	Measured	Ratio

For a dense model at short context the measured speed is 75 to 82% of the bandwidth ceiling, on hardware as different as 2017 datacenter cards, a 2020 gaming card, a 2025 gaming card and a unified-memory desktop. That is the basis for the default efficiency of 0.75. Two cases fall well short. A large mixture of experts made of many small operators reached 34% on the DGX Spark: when each token touches only a few gigabytes, fixed per-operator costs dominate. And at 224K tokens of context the dense 27B ran at 43% of a ceiling that already includes reading the cache, because attention over a long cache costs more than the bytes it reads. For those cases, lower the efficiency.

The RTX 3080 here is a card modified to 20 GB; its bandwidth is taken as the stock RTX 3080's 760 GB/s, on the assumption that the modification keeps the memory bus. The KV cache in the long-context row is assumed to be 16-bit.