How to Cut LLM API Costs With Prompt Caching

Cached input tokens can cost as little as one tenth of fresh ones, so the fastest way to cut the bill on a chat, agent, or RAG app is often not a smaller model but reusing the prompt prefix you already sent. The move is simple: put your stable content (system instructions, tool schemas, reference documents) at the front, mark it cacheable, and let the provider bill repeat reads at a fraction of the input rate. Anthropic makes you opt in with one field; OpenAI and Gemini 2.5 do it automatically once your prompt clears a size threshold.

What Do You Need Before You Start?

Three things, nothing exotic:

• An API key for Anthropic, OpenAI, or Google Gemini, and the matching SDK installed (pip install anthropic, pip install openai, or pip install google-genai).
• A prompt with a large stable portion: a long system prompt, tool or function schemas, a reference document, or a block of few-shot examples that you send on nearly every call.
• A way to read the response usage object, because caching is invisible until you inspect it.

How to Add Prompt Caching to Your API Calls

Follow these six steps in order. The first two apply to every provider; the rest are provider-specific and self-contained.

1. Sort the prompt so stable content leads. Put instructions, tool definitions, and long documents first. Keep the user's variable text, timestamps, and anything dynamic last. The cache reuses the longest identical prefix, so a single changing token near the top wastes the whole thing.
2. Confirm the prefix clears the minimum. You need at least 1,024 tokens before the cache point on Claude Opus 4.8 and Sonnet 5, at least 1,024 on OpenAI, and at least 2,048 on Gemini 2.5. Under the floor the request runs normally but caches nothing.
3. On Anthropic, set the breakpoint with cache_control. Attach it to the last block you want cached. Everything before that block becomes the cached prefix.

   import anthropic
   client = anthropic.Anthropic()
   
   resp = client.messages.create(
       model="claude-opus-4-8",
       max_tokens=1024,
       system=[
           {"type": "text", "text": LONG_SYSTEM_PROMPT},
           {"type": "text", "text": REFERENCE_DOC,
            "cache_control": {"type": "ephemeral"}},
       ],
       messages=[{"role": "user", "content": "Summarize section 4."}],
   )
   print(resp.usage)

4. On OpenAI, change nothing, then pin routing if you shard traffic. Caching applies automatically above 1,024 tokens. The optional prompt_cache_key routes similar requests to the same cache so hit rates stay high under load.

   from openai import OpenAI
   client = OpenAI()
   
   resp = client.chat.completions.create(
       model="gpt-4o",
       prompt_cache_key="rag-tenant-42",
       messages=[
           {"role": "system", "content": LONG_SYSTEM_PROMPT},
           {"role": "user", "content": "Summarize section 4."},
       ],
   )
   print(resp.usage.prompt_tokens_details)

5. On Gemini, rely on implicit caching or create an explicit cache for a guarantee. Gemini 2.5 caches automatically. When you must guarantee the discount, register the content once and reference the handle.

   from google import genai
   from google.genai import types
   
   client = genai.Client()
   cache = client.caches.create(
       model="gemini-2.5-flash",
       config=types.CreateCachedContentConfig(
           contents=[LONG_REFERENCE_DOC],
           ttl="3600s",
       ),
   )
   resp = client.models.generate_content(
       model="gemini-2.5-flash",
       contents="Summarize section 4.",
       config=types.GenerateContentConfig(cached_content=cache.name),
   )
   print(resp.usage_metadata)

6. Verify the hit before you trust it. Run the same request twice. The first call writes the cache; the second should read it. Read the counts described in the next section. If reads stay at zero, your prefix is not stable.

How the Prefix Rule Actually Works

Caching keys on an exact prefix match, not on meaning. The provider hashes your prompt from the start; the moment a byte differs from a stored request, the match ends and everything after pays full price.

That single rule drives every layout decision. Anthropic reads its prefix in a fixed order, tools then system then messages, and a change at one level invalidates that level and all the levels below it. Swap a tool definition and you lose the system and message caches too; change only the last user turn and the expensive front stays warm.

This is why non-deterministic details near the top quietly kill savings. A timestamp in the system prompt, tool schemas serialized in random key order, or a per-user greeting before the shared instructions all reset the prefix on every call.

Which Provider Caches How?

The three major APIs converge on the same idea and differ on the mechanics that matter for cost and code. This table lines up the specifics you plan around.

Read the row that matches your stack, then decide two things: is your prefix big enough to qualify, and do your calls repeat inside the cache lifetime.

When Is the 1-Hour TTL Worth Paying For?

Anthropic's default 5-minute window suits back-to-back agent turns and busy endpoints. The 1-hour TTL costs 2x the base input rate to write instead of 1.25x, so it earns its keep only when a large prefix gets reused across a gap longer than five minutes: a support session with think time between messages, or a document a user returns to over an afternoon.

Do the arithmetic before you switch. If the reused prefix is small or traffic is constant, the standard window already captures the win and the higher write price is dead weight.

What Do the Usage Fields Tell You?

Every provider reports the split so you can audit it. On Anthropic, cache_creation_input_tokens counts what got written and cache_read_input_tokens counts what came back cheap; input_tokens is the uncached remainder. On OpenAI, check usage.prompt_tokens_details.cached_tokens. On Gemini, read cached_content_token_count in usage_metadata.

Log the ratio of cached reads to total input tokens as a first-class metric. A cache hit rate that drifts toward zero is the earliest signal that a recent prompt change broke the prefix.

How Do You Fix a Cache That Never Hits?

Most caching failures trace to a handful of causes. Work through them in this order.

• Reads always show zero, writes keep happening. Your prefix changes every call. Hunt for a timestamp, a UUID, a random-ordered JSON serialization, or user-specific text sitting ahead of the shared content, and move it after the cache point.
• Nothing caches and no error appears. The cacheable region is under the token minimum. Count tokens; if you are below 1,024 (or 2,048 on Gemini 2.5), consolidate more static context ahead of the breakpoint or accept that the prompt is too small to benefit.
• Hits work in tests but vanish in production. Requests arrive too far apart and the cache expires between them. Either raise the TTL where the provider allows it, or keep the prefix warm with steady traffic.
• Anthropic writes far more than you expect. Your cache_control block sits after variable content, so the breakpoint lands too late and little gets cached. Move it to the last stable block.
• OpenAI hit rate is erratic under load. Requests scatter across cache nodes. Set a stable prompt_cache_key per tenant or session to route them together.

What to Do Next

Wire caching into how you measure and spend, not just how you call the API.

• Read the primary docs for your provider and confirm the current numbers before you commit a budget: Anthropic prompt caching, OpenAI prompt caching, and Gemini context caching.
• Instrument the cache hit rate alongside your token spend. My notes on AI token budgeting and on shrinking the token budget cover the metrics that make this visible.
• Understand why the reuse is cheap at all. The same mechanism that makes cached tokens fast is the subject of why KV cache is the biggest lever in inference cost.
• Pair caching with disciplined prompt structure. If your calls also parse model output, reliable JSON with structured outputs keeps the variable tail clean and predictable.

Sources

• Anthropic, Prompt caching (platform.claude.com docs)
• OpenAI, Prompt caching guide (developers.openai.com)
• Google, Context caching (Gemini API docs)