Token Economics: Decoding the API Bill
API calls that return identical-looking text outputs can have vastly different costs. Understanding LLM pricing requires looking past the simple prompt and looking at the accumulated context payload (history, documents, prompts) that travels with every request.
Pre-fill vs. payload: Both calls generate the same 150-word response. Call B is 87 times more expensive because the model had to reread 18,000 words of background documents and the entire chat history.
Why output tokens cost 3 to 4 times more than input tokens.
Parallel Processing
When you send a prompt, the model reads the entire sequence of input tokens simultaneously. The GPU processes them in parallel, utilizing high hardware efficiency. This step has no temporal loop dependency, allowing providers to offer lower input pricing.
Sequential Generation
Generating output is an autoregressive loop. The model predicts one token, appends it to the history, and performs a complete pass of the network to predict the next token. This sequential step is memory-bandwidth bound, requiring high computational overhead per token.
The growing tax of chat history and dynamic retrievals.
System Schemas
Instructions and tool definitions (such as JSON API declarations) are sent with every request. While static, a bloated system schema acts as a permanent flat tax on every single execution cycle.
Chat Accumulation
In multi-turn chat agents, the entire conversation history must be sent with every message. If the history grows by 500 tokens per turn, request 10 carries 5,000 tokens of overhead, inflating costs linearly.
RAG Payloads
Retrieval-Augmented Generation injects raw database query rows, code logs, or documentation pages. Injecting a single 10,000-token file to answer a simple query can instantly increase the bill by 100x compared to base prompting.
Prompt Caching: reducing static input costs by up to 90%.
Modern API providers support prompt caching. If a request shares a long prefix with a previous request (such as a large system prompt, standard codebase conventions, or fixed documentation files), the model retrieves the pre-processed sequence from cache rather than recalculating it.
Uncached Prefix
Every request reads the entire system prompt, tool definitions, and RAG document from scratch. You pay 100% of the standard input rate on every turn, regardless of repeating context.
Cached Prefix Match
The repeating system schemas and RAG references are matched in memory. You only pay a minor cache-write fee on hit one, and a 90% discounted read fee on subsequent matching calls.
Four strategies to keep context payloads lean.
Sliding Window History
Do not append every turn indefinitely. Implement a sliding context window that only keeps the last 5-10 messages, purging older turns to cap the linear cost growth.
RAG Retrieval Limits
Restrict chunk injection. Set strict limits on the number of returned database rows and document blocks injected per query (e.g. max 3 chunks or 2,000 tokens of reference material).
Compact System Schemas
Remove unused helper explanations and optimize JSON schemas. Strip whitespace and shorten descriptions to ensure your static prompt footprint remains minimal.
Hard Output Caps
Set max token limits on generation parameters. A rogue model loop or formatting mistake can write infinite loops, costing dollars per request if not bounded by the client.
"Cost optimization in AI is the discipline of managing context, not just models."
Input tokens are cheap but cumulative. Chat history and dynamic documents will quietly inflate your bills if left unbounded.
Align prompts to leverage caching. Order your requests so that static content remains at the start of the prompt to maximize cache hits.