HopBaseMost users checking their first bill ask: why is someone else's cost so much lower for the same model? The answer, eight times out of ten, is prompt cache. This article breaks down the billing mechanics, real-world hit rates, and how to structure prompts to capture the full discount.
Billing Mechanics: Cached Prefix Reads Are Heavily Discounted
LLM API input billing has two tiers: first-pass tokens cost the standard rate; cached prefix hits cost a steep discount (for Claude, cached reads are one-tenth the standard input rate). Put plainly: the more stable your session prefix and the longer the repeated section, the lower your effective input cost.
Real-World Hit Rates
From our gateway vantage point: conversational tools like Claude Code and AI Chat carry system prompts and message history every turn, so healthy sessions typically see cache hit rates above 60%. Poor implementations — say, embedding timestamps in the system prompt — can zero out caching entirely, multiplying costs several times over.
Three Techniques to Double Your Hit Rate
- Put "never changing" first, "changing" last.Cache matches by prefix: system prompt, tool definitions, few-shot examples go at the start; user input and retrieved results go at the end. A dynamic field in the middle invalidates everything after it.
- Never put dynamic data in your prompt.Timestamps, random IDs, "what day is it" — these are cache killers. If you need a timestamp, put it in the user's message at the end.
- Continue sessions, don't restart them.Keep multi-turn interactions in the same session to accumulate cache hits. Restarting for each turn burns the cached prefix every time.
How to Verify You're Getting the Discount
Your console's usage details break out standard input, cache write, and cache read tokens for each request. After reorganizing your prompts, run a few turns and compare the cache-read share — whether your optimization worked shows up in numbers instantly.
Caching isn't magic, it's prefix matching. Organize your prompts as "immutable prefix + mutable suffix," and your bill will honestly reward you.