TL;DR: Contain runaway requests immediately with step, retry and spend limits, while measuring cost per successful business outcome. Then evaluate caching, context size and model routing against quality. Per-tenant and per-feature budgets need defined behavior on exhaustion.
How to approach it
Check whether spend growth reflects useful demand, a price change or amplification such as retries. Stop an uncontrolled loop before waiting for a complete attribution project. Then prioritize optimizations by measured savings and quality risk.
A strong answer
Instrument before you cut. Tokens in and out, per request, tagged by feature, tenant, model and call site. Chain steps counted individually, because a single user action that makes four model calls looks like one request in an application metric. With that, cost per unit falls out: per resolved support ticket, per generated summary, per document indexed. That number is what tells you whether this is growth or waste, and it is the number a finance conversation needs.
The usual causes of a sudden multiple, in the order they tend to appear:
- Retries and agent loops. A retry on a failed parse doubles cost for that request. An agent that loops until it succeeds has no ceiling at all, which is why a maximum step count is a cost control and not only a safety one.
- Context growth. Retrieval returning ten documents instead of three, a system prompt that grew through additions, conversation history sent in full every turn. Compare input, cached-input and output token counts multiplied by their actual rates; volume alone does not identify the largest cost.
- A model change. Someone moved to a larger model for quality and the price per token changed by an order of magnitude.
- A new caller. A batch job or an internal tool that started using the endpoint, which no product metric shows.
The levers, best ratio first:
| Lever | Typical saving | Risk |
|---|---|---|
| Prompt and semantic caching | Large on repetitive traffic | Staleness, tenant leakage if mis-keyed |
| Trim context and history | Large, often the biggest | Quality loss if trimmed blindly |
| Route by task difficulty | Large | Needs a quality floor and monitoring |
| Batch the offline paths | Substantial on async work | Latency, unsuitable for interactive |
| Cap steps, outputs, retries | Bounded but immediate | Truncation, visible to users |
| Per-tenant limits | Protects the worst case | Customer friction |
Evaluate caching by cache type. Exact response caches can become stale and freeze a previously sampled answer. Prefix KV caching reuses computation for identical prefixes; semantic response caching can return an answer to a different question and needs quality evaluation. Include tenant, authorization scope, model version and relevant data freshness in response-cache keys.
Routing is the largest structural win and needs the most care. Send the easy majority to a smaller model, escalate to the large one on a confidence signal or a validation failure, and monitor the escalation rate as a quality SLI. Done without that monitoring, it is a silent quality cut.
Then make the ceiling structural. Per-tenant and per-feature token budgets enforced at the gateway, with alerts at a burn-rate threshold rather than at the limit, and a defined behaviour on exhaustion: degrade, queue, or refuse with a clear message. A budget discovered at the invoice is not a control.
What interviewers probe next
"Which cost is bigger, input or output?" Compute each priced component. For an illustrative request with 4,000 input tokens at $1/million and 1,000 output tokens at $10/million, input costs $0.004 and output costs $0.01. Include cached-input and tool charges separately.
"How do you decide a feature is not worth its cost?" Cost per unit against the value of that unit, with the same threshold you would apply to any other margin decision. Having the number is what makes the conversation possible.
"What about a customer whose usage is unprofitable?" Price it or limit it, and know which before the contract renews. A flat-rate AI feature with unbounded usage is a margin problem waiting to compound.
Common mistakes
Cutting output limits without measuring output cost and completion quality; emergency containment still takes priority over optimization.
No per-feature attribution, so the only available action is turning something off.
Model routing with no quality monitoring, which saves money now and produces a churn conversation two quarters later.