The cost reviews I sit in start from the wrong end of the bill. Someone has a monthly number that has grown too large, and the first proposals are always about generation: cap the output length, ask the model for terser answers, move to a smaller model that writes less. Every one of those attacks the part of the bill I can usually show is a few percent of the total. In the systems I build, the model spends far more of its budget reading than writing, and most of what it reads is the same context sent to it again on the next step.
So I have stopped treating a large language model (LLM) bill as a list of switches to flip and started treating it as a product of a few numbers multiplied together. Cost is not one quantity you shave. It is calls times tokens times price, and the tokens split into a side you read and a side you write, at different prices. Once the bill is written that way, most cost questions answer themselves, because a product is dominated by its largest factor, and shaving any of the smaller ones changes almost nothing.
The useful work is finding which factor dominates your workload, and then spending your effort there. For most of what people are building now, the answer is the same, and it is not the one the generation-first instinct points at.
The bill is a product, not a list
Write the cost of a single request out in full and there is not much to it:
cost = input_tokens × input_price
+ output_tokens × output_price
The cost of a whole feature is that per-request cost multiplied by how many requests it makes. Four numbers, one of them counted twice. Everything sold as a cost technique is an attempt to push down one of these four factors, and knowing which factor a technique touches tells you almost everything about whether it is worth doing.
Start with the two prices, because they are fixed and public and they set an intuition that turns out to be a trap. Output tokens cost several times what input tokens cost, everywhere. Anthropic prices its current line at a flat five to one: Claude Opus 5 is 5 dollars per million input tokens and 25 dollars per million output, Claude Sonnet 5 is 2 and 10, Claude Haiku 4.5 is 1 and 5. OpenAI's ratio is wider and less regular: GPT-5 is 1.25 dollars per million input and 10 output, which is eight to one, while GPT-4o is 2.50 and 10, which is four to one.
Read those numbers on their own and the generation-first instinct looks correct. A token you write costs four to eight times a token you read, so the obvious lever is to write fewer of them. That is where output caps, terser-answer instructions and structured outputs come from, and it is why they are usually the first thing proposed. The instinct is right about the price and wrong about the bill, because price is only one of the two factors that decide what output actually costs you, and it is the smaller one.
Why the reading side wins, and why agents make it worse
The factor the price hides is volume. A production request almost never carries just a question. It carries a system prompt, a set of tool definitions, some few-shot examples, whatever a retriever pulled back, and the conversation so far. The answer that comes back is often a few hundred tokens. So a request can easily read tens of thousands of input tokens to write a few hundred output ones, and a ratio of fifty or a hundred to one on volume swamps a ratio of four to eight on price. Output is the expensive side per token and the cheap side per request, and it is the per-request number that lands on the invoice. The retrieved context is a large and often padded part of that per-request number, and retrieving fewer, better-ordered passages lowers it without hurting the answer.
That is the static picture, for one call. Agents make it dynamic, and the dynamics are where the bill really escapes. An LLM API is stateless: it remembers nothing between calls, so every turn resends the entire conversation so far. A tool-using agent that takes ten steps does not send its context ten times. On step one it sends the context once, on step two it sends step one's context plus the first result, on step three all of that plus the second result, and so on. The input it pays for grows with the square of the number of steps, not linearly with it. A twenty-step agent pays for something closer to two hundred context-reads than twenty.
This is the same arithmetic that makes self-critique loops expensive, and it compounds with them. A critic loop adds calls, each call resends a longer transcript, and the two effects multiply. I have argued separately that most self-correction loops spend several times the tokens to reach a worse answer; the cost half of that argument is this resend curve. Every extra pass round a loop is not one more call, it is one more call at the current, longest, context length.
Once you see the bill this way, the ranking of techniques inverts. The generation-side levers act on the output term, which is the small one. The techniques that matter act on the input term, and specifically on the part of it that is resent unchanged. The single largest one is not an optimisation you build at all. It is a discount the providers already offer.
Three caches, three different bets
Caching is where the confusion is thickest, because three unrelated mechanisms share the word and get listed side by side as though they were variations on one idea. They are not. They run in different places, match on different things, and one of them can return a wrong answer while the other two cannot.
Prompt caching is the provider's own. You mark a prefix of your request, and on the next request that starts with the same bytes the provider serves that prefix from its cache instead of recomputing it. Anthropic charges about a tenth of the normal input rate for a cached read, against a one-time write premium of 1.25 times for the default five-minute lifetime or twice for a one-hour one. OpenAI applies the same idea automatically to any prompt over 1,024 tokens and discounts the cached portion by 50 to 90 percent depending on the model. This is the technique that attacks the dominant term directly: the resent prefix in an agent loop, the fixed system prompt and tool definitions on every call, the retrieved document you ask three questions about. It is lossless, it needs almost no code, and it is the first thing I turn on. It is also what flattens the agent resend curve above, which is why the two halves of this argument are really one.
Response caching is yours to build, and it is the oldest idea in the book: store the answer to a request, and if the exact same request comes in again, return the stored answer without calling the model. It is lossless, because the thing you cached is a real answer the model actually gave. Its limit is the hit rate. Exact repeats are common for narrow, closed inputs (a fixed classification prompt over a small set of documents) and rare for anything a human types freely, because two people almost never phrase a question identically.
Semantic caching tries to rescue that hit rate, and it is the one to be careful with. Instead of matching the request exactly, it embeds the request and returns a stored answer if some previous request is close enough in embedding space. That raises the hit rate by design, and it introduces a failure the other two caches cannot have: it can serve the answer to a different question. The MeanCache study measured this on a widely used semantic cache, GPTCache, and found it produced 233 false hits across 700 unique queries at a precision of 0.52. Roughly one unique query in three got an answer that belonged to a different query. The dominant failure is negation: "how do I enable two-factor authentication" and "how do I disable two-factor authentication" sit above 0.95 similarity in most embedding models, because the shared nouns dominate the vector and the one word that reverses the meaning barely moves it. A semantic cache set loose on those returns one for the other.
| Cache | Match | Lossless? | Main risk |
|---|---|---|---|
| Prompt | Exact prefix | Yes | Breaks on any prefix change |
| Response | Exact request | Yes | Low hit rate on free text |
| Semantic | Similarity | No | Can serve a wrong answer |
The table runs provider-side at the top and in your own application below. It has a diagonal to it. As you move down, the hit rate goes up and the safety goes down. Prompt caching is a pure win with a low ceiling on what it can match. Semantic caching matches far more and can be wrong. That is not an argument against semantic caching everywhere; it is an argument for treating it as a correctness decision that happens to save money, gated by a similarity threshold you have actually measured against your own queries, rather than as a cost technique you switch on. The threshold that looks safe on a demo is the one that flips "disable" for "enable" in production.
The optimisations that cancel each other
The trouble with a long list of separate techniques is that it invites you to apply several at once, and some of them fight. The fights are all about the same thing: prompt caching depends on the front of your request never changing, and several popular token-savers work by changing it.
Prompt caching is a prefix match. Any byte that changes anywhere in the cached prefix invalidates the cache from that point on, and you pay the full read rate again plus the write premium to re-establish it. The providers render a request in a fixed order (tools, then system prompt, then messages), so the stable material has to sit at the front and the volatile material at the back. This makes the order of your prompt a cost variable, which it was not before caching existed.
Now look at two of the most-recommended token-savers against that constraint. Context trimming, the practice of dropping older or less relevant history to shrink the request, edits the middle of the context and reshuffles what follows it, invalidating the cache behind the edit. Conversation summarisation, which replaces a long history with a shorter synopsis, rewrites the prefix outright every time it fires. Both genuinely reduce the token count of the request in isolation. Both, applied on top of prompt caching, can raise the bill, because they turn a stream of tenth-price cached reads into full-price reads plus write premiums. Whether they are worth it is an arithmetic question specific to your traffic, and it is the opposite of the arithmetic you would do if you thought of them as free.
Multi-model routing has the same problem from a different angle. Caches are scoped to a model: a prefix cached against one model is not available to another. Route a call to a different model to save on its per-token price and you get a cold cache, so the saving on the price has to beat the loss of the discount on the volume. For the input-heavy workloads this whole piece is about, that trade often goes the wrong way, and a cascade of models that each look cheaper in isolation can cost more than staying on one model with a warm cache. The same logic applies to switching effort levels or other request settings mid-conversation if doing so invalidates the cache: the cache is usually worth more than the setting.
The token-savers are worth having; they are simply not independent, and the thing they most often conflict with is the single most valuable lever you have. Turn on caching first, structure the prompt so the cache survives, and then decide whether trimming or routing still pays once the discount they displace is priced in.
The levers that cost quality, and the ones that are not levers
Two techniques on every such list do reliably reduce cost and are worth having, with a condition attached. Smaller models, and their cousins model routing and distillation, cut the per-token price, sometimes by a lot: the Amazon Web Services (AWS) guidance I have written about elsewhere reports Bedrock's own claims of up to 30 percent from intelligent routing and models up to 75 percent cheaper from distillation with under 2 percent accuracy loss on retrieval-shaped tasks. Those are vendor numbers and deserve the usual scepticism, but the mechanism is real and the direction is right.
The condition is that these levers spend quality, and you cannot price the trade without measuring it. A smaller model is cheaper on every request and worse on some fraction of them, and which fraction is a property of your task that no pricing table can tell you. This is where a cost programme meets an evaluation programme: the only thing that licenses moving a route to a cheaper model is a test set that says the cheaper model still passes on that route. Picking the smallest model whose built-in assumptions match the problem is usually the more accurate choice as well as the cheaper one, but "usually" is doing real work in that sentence, and the eval is how you find the exceptions before your users do.
Batch processing belongs in the free-win column with caching, for the narrow set of work that can tolerate it. Both major providers run an asynchronous batch tier at half the synchronous price, in exchange for a turnaround measured in hours. For anything not in a user's request path (overnight enrichment, backfilling embeddings, generating an evaluation set) that is a 50 percent discount for accepting latency you did not need to avoid. It is lossless and it competes with nothing.
Then there are two things that appear on cost lists and are not cost techniques at all, which is worth saying plainly because treating them as levers wastes effort. A monitoring dashboard does not reduce a bill; it tells you which of the four factors is large, which is the necessary first step and not a saving in itself. Every cost decision above depends on knowing your own token profile, and the dashboard is how you know it, so it earns its place, just not in the column people file it under. Agent guardrails (a cap on iterations, a cap on tokens, a timeout) are the same kind of thing one level up: they bound the worst case, stopping a runaway loop from generating a five-figure surprise, but they do nothing to the cost of the normal request. They are insurance against the tail, not a reduction of the mean.
The unifying discipline across all of these is to count cost per completed task, not per request. A cheaper request that fails more often and has to be retried, or that comes from a smaller model that needs two passes to get the answer a larger one gave in one, is not cheaper. The invoice is per token; the thing you actually care about is per outcome, and the two only line up when the quality holds.
What I would defend, and what I would not
Three things I would defend without hedging. Write your bill as a product and find its dominant factor before you optimise anything, because effort spent on a small factor is wasted no matter how clever it is. Take the lossless discounts first, caching and batch, because they cost quality nothing and caching in particular acts on the factor that usually dominates. And treat every quality-trading lever, from smaller models to semantic caching, as gated on a measurement, because the entire value of the saving depends on a quality claim you have not checked until you check it.
One thing I hold more loosely, and it is the boundary of the whole argument. Input dominance is a property of a workload, not a law. A feature that writes long documents from short prompts spends its budget on output, and the generation-side levers this piece is sceptical of are exactly the right ones there. Reasoning models complicate it further, because their hidden reasoning is output, it is billed, and it is growing: as more of a model's work moves into tokens it generates before it answers, the output term grows relative to the input one, and a workload that is input-bound today could be output-bound on next year's models. The method survives that shift even where the conclusion does not. Write the bill as a product, measure which factor is largest for the traffic you actually have, and spend your effort there. The factor that dominates is the one worth arguing about, and it is knowable from your own logs long before it is knowable from a list.