Prompt caching is the most underused cost lever in production AI. It can cut a bill by more than half without changing the model or the output.
It also breaks silently if you structure prompts the wrong way. This guide covers both.
For current model prices, see latest AI models. For the cost basics, see AI model costs explained.
What prompt caching is
Providers can store the processed form of a prompt prefix and reuse it on the next request. Instead of paying full input price for the same system prompt, tool schema and documents, you pay a much lower cache-read price.
The trade-off: you usually pay a small premium to write the cache, and the cache expires after a period of inactivity.
| Token type | Billed at | When it applies |
|---|---|---|
| Fresh input | Full input price | First time the prefix is processed |
| Cache write | Often around 1.25× input | When the prefix is stored |
| Cache read | A fraction of input | When the prefix is reused |
Exact rates and expiry windows differ by provider. Check the current price for the model you use.
Why it matters for agents and RAG
An agent resends its system prompt, tool definitions, retrieved documents and conversation history on every turn. On a long run, most input tokens are repeats.
That makes the cache-read rate, not the headline input price, the number that decides your bill.
A worked example
Take 100 million input tokens and 10 million output tokens a month, with 90% of input cached, on Claude Fable 5.1 at $10 input / $50 output / $0.25 cached read.
| Line item | No caching | With caching |
|---|---|---|
| Cached input (90M) | $900 | $22.50 |
| Fresh input (10M) | $100 | $100 |
| Output (10M) | $500 | $500 |
| Total | $1,500 | $622.50 |
That is a saving of about 58% on the same workload. At 95% cached, it is closer to 62%.
The saving scales with two things: how much of your prompt is repeated, and how much output you generate. Output is never cached.
What breaks caching
Caching works on a prefix. The moment the prefix changes, the cache misses.
Common mistakes:
- A timestamp or request ID at the top of the system prompt. Every request becomes unique.
- User-specific data placed before shared context. Put stable content first, variable content last.
- A tool list that changes order. Sort tools deterministically.
- Rebuilding the system prompt from a template with random ordering.
- Injecting the current date, user name or session ID into the first paragraph.
- Very short prompts. If the prefix is small, the cache-write premium can outweigh the saving.
Design rules
- Stable content first. System instructions, policies, tool schemas and reference documents go at the top.
- Variable content last. The user question, session details and fresh data go at the end.
- Keep prefixes identical. Same text, same order, same whitespace.
- Cache the expensive parts. Long documents and tool definitions are the best candidates.
- Measure the hit rate. If it is below 50%, the prompt structure is the problem.
- Watch the write cost. A cache write on a prefix that is never reused is wasted money.
What to cache
| Content | Cache? | Why |
|---|---|---|
| System prompt | Yes | Same every request |
| Tool definitions | Yes | Large and stable |
| Brand or policy documents | Yes | Reused across sessions |
| Retrieved documents | Sometimes | Only if reused across turns |
| User question | No | Unique each time |
| Current date | No | Changes daily; put it at the end |
| Session history | Depends | Useful in long agents, expensive if short |
How to measure it
Track:
- Cache hit rate by endpoint.
- Cache write tokens and their cost.
- Cost per request before and after.
- Latency, which often improves too.
A hit rate above 70% on an agent workload is usually a sign the prompt is structured well. Below 30% usually means a variable field is too early in the prompt.
When caching is not worth it
- One-off requests with no repeated prefix.
- Very short prompts where the write premium dominates.
- Workloads with highly variable context on every call.
- Providers that do not offer caching for your model.
In those cases, batch pricing or a cheaper model will save more.
FAQ
What is prompt caching?
A provider feature that stores a repeated prompt prefix and bills reuse at a lower rate than fresh input.
Does caching change the model’s answer?
No. It changes how the input is processed and billed, not the output.
Why is my cache hit rate low?
Usually because a variable field, timestamp or reordered tool list appears in the cached prefix. Move variable content to the end.
Is caching better than switching to a cheaper model?
They solve different problems. Caching reduces the cost of repeated context; a cheaper model reduces the cost of everything. Use both.
Next step
If you are running agents or RAG in production and want the cost model reviewed, contact 24Bit System. Our Fable 5.1 notes cover the cache pricing change, and AI model costs explained covers the rest.
