Two teams can use the same model and receive very different bills. The difference is rarely the headline price per million tokens. It is how the requests are built, retried and cached.
Understanding the billing units is the first step to a realistic AI budget.
Input and output tokens
Every request has an input and an output. The input includes the system prompt, the conversation history, retrieved documents and the user message. The output includes the answer and any reasoning tokens the model generates.
Output usually costs more than input. A long answer with a large context can cost several times a short answer with the same question.
Reasoning tokens
Reasoning models may generate internal tokens before the visible answer. Those tokens are billed even when the user never sees them. A model that “thinks” for a long time can be expensive on simple tasks.
Set a reasoning effort appropriate to the task. Not every support reply needs deep reasoning.
Cache reads and writes
Providers charge differently for cached context. A cache write may cost more than a normal input token; a cache read may cost much less. If the same long instruction or document is sent repeatedly, caching can cut the bill sharply.
Caching only helps when the prefix is stable. Changing the system prompt or the document order on every request defeats it.
Batch pricing
Batch APIs trade latency for a discount. They are useful for offline jobs: classifying a month of tickets, summarizing a document archive, enriching a product catalogue.
Interactive chat should not use batch. The discount is not worth a delayed answer.
Retries and context growth
A failed request can still be billed, depending on the provider and the failure. Retries add cost, especially when the model times out after generating most of the answer.
Conversation history is another hidden cost. A chat that keeps the full transcript grows with every turn. Summarize older turns or retrieve only the relevant part.
A cost model example
Take a support assistant that handles 10,000 conversations a month. Each conversation has a 2,000-token instruction and retrieved context, a 300-token answer and two retries on 10 percent of requests.
The exact number depends on the provider, but the structure is what matters: input volume, output volume, retry rate, cache hit rate and human review time. Model the ranges, not a single number.
How to reduce the bill
Shorten the prompt. Retrieve fewer, better passages. Cache the stable prefix. Route simple requests to a smaller model. Use batch for offline work. Measure cost per resolved task, not cost per token.
The cheapest model is not always the cheapest system. A slightly more expensive model that answers correctly on the first attempt can cost less than a cheap model that needs three tries.
A budget worksheet
Build a simple worksheet before the pilot. List the monthly request volume, the average input tokens, the average output tokens, the retry rate and the cache hit rate. Multiply by the provider price for the tier you will use.
Add the non-token costs: retrieval, storage, observability, evaluation runs and human review. For a support assistant, human review is often the largest line.
Model three scenarios: current volume, double volume and a bad month with a higher retry rate. The bad month is the one that breaks a budget approved on the average.
Review the worksheet after launch with actual usage. The first month usually reveals a prompt or a retry pattern that the estimate missed.
Forecasting the first month
Estimate the first month with a small pilot, not a full rollout. Run 100 to 200 real requests and record the token counts, retries, latency and human edits.
Multiply the observed usage by the expected volume, then add a margin for growth and bad requests. A pilot with clean inputs will understate the real bill.
Compare the estimate with the provider’s minimum commitment. A low usage pilot can sit below a monthly minimum, which makes the unit cost look better than it will at scale.
FAQ
Why is our AI bill higher than the token estimate?
Common causes are long prompts, full conversation history, retries, reasoning tokens and cache misses. Measure the actual request shape before changing models.
Does caching always reduce cost?
Only when the cached prefix is reused. A prompt that changes on every request will not benefit.
Should we use batch for everything?
No. Batch is for offline jobs where a delay is acceptable. Interactive features need a real-time response.
For build and budget planning, see AI app development cost and the AI model comparison. For a custom integration, start with custom software development.
