Inference is usually priced per token, which is convenient and conceals almost everything that determines the actual cost. Two requests with identical token counts can differ several fold in what they consume, and understanding why is the difference between an estimate and a guess.
Two phases with opposite bottlenecks
Generating a response happens in two distinct stages, and they stress different parts of the hardware.
| Phase | What happens | Limited by |
|---|---|---|
| Processing the input | Every prompt token handled in parallel | Compute |
| Generating the output | One token at a time, each depending on the last | Memory bandwidth |
This asymmetry is why output tokens are typically priced higher than input tokens. It is not a margin decision, producing a token costs meaningfully more than reading one.
The cache that decides your concurrency
To avoid recomputing the entire sequence for every new token, the intermediate attention state for all preceding tokens is retained in memory. That store grows with the length of the sequence, with the number of layers in the model, and with every concurrent request being served.
Its size is the practical limit on how many users a given accelerator can handle simultaneously. The model weights are a fixed cost, loaded once, but this cache is a per request cost that grows as each conversation lengthens. A server comfortably handling many short conversations will handle far fewer long ones, on identical hardware, with no change to the model.
This also explains why quantisation helps less than expected in production. Reducing the precision of the weights shrinks the fixed cost and frees memory, but the per request cache is unaffected, so the ceiling on concurrency moves less than the headline memory saving suggests.
Batching, and the trade it forces
Because generation is limited by memory bandwidth, processing several requests together is close to free in time. The weights are read once and applied to every request in the batch, so throughput rises substantially while the time per step barely moves.
The cost is latency for the individual request, which now waits to be grouped and shares its step with others. This is the fundamental operating trade in inference serving. Optimise for throughput and cost per request falls while each user waits longer. Optimise for latency and the reverse.
It is also why utilisation dominates the economics of self hosting. An accelerator costs the same whether it is saturated or idle, so the comparison against a metered service is not a comparison of prices, it is a question of how consistently the hardware is busy. Steady high volume favours owning it. Intermittent or unpredictable volume rarely does.
What has to fit in memory
Three things occupy accelerator memory simultaneously, and a deployment fails if their sum exceeds what is available.
The weights are the largest single item and are fixed once the model and its precision are chosen. The attention cache described above is per request and grows with conversation length. Activations, the working values produced during a forward pass, are transient but non trivial, particularly at large batch sizes.
The practical consequence is that model choice and concurrency are the same decision rather than two. A larger model leaves less room for the cache, which means fewer simultaneous conversations or shorter ones. Fitting a bigger model onto the same hardware is not a success if it halves how many people it can serve.
Model size is a cost decision, not a quality decision
The instinct is to select the most capable model and use it everywhere, which is usually the most expensive way to build a system that is no better.
Most workloads contain a distribution of difficulty. Classification, extraction, routing, and reformatting are handled well by small models, while a minority of genuinely hard reasoning steps benefit from a large one. Routing by difficulty, so that the expensive model handles only the cases that need it, commonly reduces spend substantially with no measurable loss in outcome.
Establishing that requires an evaluation set, which is the recurring theme. Without one, model choice is decided by impression, and impression consistently favours the largest available option.
Why long context costs more than its token count suggests
The attention mechanism relates every token to every other, so processing an input grows faster than linearly with its length. Doubling the prompt more than doubles the work of that first phase, even before any output is produced.
The effect compounds in agentic loops. If the accumulated history is resubmitted on every iteration, the same context is processed again at every step, and a loop of twenty steps over a growing context is doing a great deal of repeated work. Cost then scales with the product of loop length and context size, which is a much steeper curve than either alone.
Caching the processed form of a repeated prefix addresses exactly this, and where it applies the saving is large. It only applies to a genuinely unchanged prefix, which is a reason to structure the stable parts of a prompt first and the variable parts last.
The costs that do not appear on the invoice
Token spend is usually the smaller half of the total.
Failed and retried calls are paid for in full. Evaluation, if it is done properly, means running representative tasks repeatedly on every change, and a serious evaluation suite can cost more to operate than production traffic. Human review of output is the largest cost in many deployments and is invariably accounted for somewhere other than the inference budget. And a system that answers plausibly but wrongly imposes a cost that never appears in any ledger at all.
Note: the two figures that predict cost better than token counts are how long the average conversation becomes and how consistently the hardware is busy. Both are properties of the product rather than the model.