Tokens are the unit of cost, latency and capacity at once

TL;DR: Two requests per second can differ tenfold in cost and duration depending on their token counts. Instrument tokens in and out per request, tagged by feature and tenant, before optimising anything. Compute the bill from input, output and cached-token rates; generation length also affects response duration.

Why request count stops working

One request sends 200 tokens and generates 50. The next sends 30,000 and generates 2,000. In an access log they are two identical lines. In cost, in duration and in how much of the GPU they occupy they are two orders of magnitude apart.

So any dashboard, autoscaler, rate limiter or capacity model built on request count is measuring something adjacent to load rather than load itself. A rate limit of 100 requests per minute permits somewhere between a trivial workload and one that saturates the hardware, and which of those arrives is decided by the callers rather than by you.

This is a difference of degree rather than of kind, and the degree is what matters. Ordinary web requests vary too: a cached page and a complex search are not the same work. But the spread is usually within an order of magnitude and roughly bounded by the endpoint, whereas a model endpoint's cost is set by an argument the caller supplies. Token counts are the measure that makes that variation visible, used alongside request count rather than instead of it.

The three things tokens decide

Cost. Providers price input and output separately, output typically several times higher per token. Self-hosting converts this into GPU-hours, but the unit that divides the bill is still tokens.

Latency. Input tokens are processed in parallel during prefill, so they mostly affect time to first token. Output tokens are produced one at a time, so they set the total duration almost linearly. If per-token decode time is unchanged, 2,000 output tokens require roughly twenty times the decode duration of 100; total latency also includes prefill and queueing.

Capacity. Every token of every active sequence occupies KV cache memory, and that memory is what limits concurrency. Longer conversations mean fewer of them at once.

One unit, three consequences, which is why these stop being separate conversations.

The asymmetry people get backwards

Use the selected model's prices. For illustrative rates of $1 per million input tokens and $10 per million output tokens, 4,000 input tokens cost $0.004 while 1,000 output tokens cost $0.01. More input tokens does not imply a larger input bill.

The practical consequence is that teams optimise the visible thing (capping maximum output length, which users notice) and leave a 4,000-token context untouched. Measure both before choosing.

Where the multiplication happens

Token spend has no natural ceiling, and it compounds in specific places:

• Retries. A retry adds another attempt; token lengths, caching and where the first attempt failed determine the extra charge.
• Agent loops. A loop that runs until it succeeds has no bound at all. A maximum step count is a cost control before it is a safety one.
• Chains. Five sequential calls means five prompts, five contexts, five latency draws and five chances to fail.
• History. Sending the full conversation every turn makes cost grow quadratically over a long session.
• Retrieval. Raising the number of retrieved documents from 3 to 10 multiplies the retrieved portion of the prompt by 10 / 3 ≈ 3.3, assuming documents of similar length. Total input grows by less than that, because the fixed instructions and the conversation history do not change, which is also why the effect is easy to miss in a prompt-length average.

One feature, costed all the way through

The argument is easier to trust once the arithmetic has been done once, so take a feature. A support tool summarises conversations: 40,000 conversations a month, each sending about 3,500 input tokens and generating about 400. Use the illustrative rates above, $1 per million input tokens and $10 per million output tokens; substitute your model's real prices, which are the only ones that matter.

Input is 40,000 × 3,500 = 140,000,000 tokens, which is 140 million, so 140 × $1 = $140. Output is 40,000 × 400 = 16,000,000 tokens, so 16 × $10 = $160. The feature costs about $140 + $160 = $300 a month, and the unit figure is $300 / 40,000 = $0.0075 per conversation.

Look at which half is larger. Input is 3,500 / 400 = 8.75 times more tokens than output and costs less than output does, because the output rate is ten times higher. That is the asymmetry above, now with a number on it, and it is why capping output length is the optimisation teams reach for and rarely the largest one available.

Now change one thing that nobody files as a cost change. Somebody adds a 2,000-token instruction block to the prompt to improve tone. Input per conversation becomes 5,500, so input tokens are 40,000 × 5,500 = 220,000,000 and input cost becomes $220. The bill goes from $300 to $220 + $160 = $380, a rise of 380 / 300 ≈ 1.27, or 27 percent, from a prompt edit in a pull request that changed no code path and no configuration.

What this does not establish is the quality effect of either the instruction block or its removal, and that is the point at which the arithmetic hands over rather than deciding. It also assumes no prompt caching; a cached prefix is billed at a different rate and would change the input half substantially. The calculation's job is to make the prompt edit visible as a 27 percent cost decision, so that somebody weighs it.

The number worth computing

Total spend is unmanageable. Cost per unit of business value (per resolved ticket, per document processed, per summary generated) is a number a team can act on and a finance conversation can use. It also answers the question that matters when a bill grows: eight times the spend alongside eight times the volume has flat unit cost, which still needs quality and revenue checks, and eight times the spend on flat volume is a defect.

Instrument before optimising

Tokens in and out, per request, tagged by feature, tenant, model and call site, with chain steps counted individually. Without that, the only available lever is turning a feature off, which is the decision a team is forced into when a bill arrives that nobody can divide.

Self-check

A chat feature handles 10,000 sessions a month. Each session runs 8 turns, and the full conversation is resent every turn. The first user message is 300 tokens, each reply is 250 output tokens, and a fixed system prompt of 700 tokens is sent every turn. Estimate the input tokens one session consumes, say which mechanism from the list above dominates, and then decide which of two changes saves more: trimming the system prompt to 200 tokens, or capping replies at 150 tokens. Use $1 per million input and $10 per million output. Work it through before reading on.

Take the input side turn by turn, since the conversation grows as it goes. Turn one sends the system prompt plus the first message, 700 + 300 = 1,000. Turn two sends the same plus the first reply and the next user message; with 250-token replies and 300-token messages the history grows by about 550 tokens a turn. So the turns send roughly 1,000, then 1,550, 2,100, 2,650, 3,200, 3,750, 4,300 and 4,850, and the total is about 23,400 input tokens for one session. Output is 8 × 250 = 2,000 tokens.

The mechanism is history. The system prompt contributes 8 × 700 = 5,600 of those tokens and the conversation replay contributes most of the rest, which is the quadratic growth the list warns about: each turn pays again for every turn before it.

Now the two changes. Trimming the system prompt from 700 to 200 saves 500 tokens on each of 8 turns, so 8 × 500 = 4,000 input tokens a session, or 10,000 × 4,000 = 40,000,000 a month, which at $1 per million is $40.

Capping replies at 150 looks smaller and is not. It saves 8 × 100 = 800 output tokens directly, worth 10,000 × 800 = 8,000,000 output tokens, or $80 at $10 per million. It also shortens every subsequent replay of those replies, and that resent history is billed as input: a reply that is 100 tokens shorter is resent on each later turn, saving roughly 100 × (7 + 6 + 5 + 4 + 3 + 2 + 1) = 2,800 input tokens a session, another $28 a month. So the output cap saves about $108 against the prompt trim's $40.

The reasoning worth keeping is that the reply cap wins on both sides of the ledger at once, because output tokens are billed at the higher rate and become input tokens on every later turn. And the limit worth stating: shorter replies are a product change that users notice, whereas a shorter system prompt is invisible to them, so the cheaper saving may still be the one to make first.

Next: the KV cache explains why those same tokens also cap how many sessions fit on one GPU, and unit economics turns a monthly figure into one a team can act on.