Semantic Caching for LLMs: Cost, Accuracy, and Risk

AWS engineers cut large language model inference cost by 86.3% on a real chatbot workload, not by switching models or renegotiating a rate card, but by answering repeat questions from a cache instead of calling the model at all. The technique, semantic caching, now sits alongside prompt caching and model routing as one of the highest-return cost levers in production. It also introduces a failure mode the other two do not: a cache that returns a confidently wrong answer because two questions looked similar and were not.

This piece covers what semantic caching saves, the single knob that governs both the savings and the risk, and why a technique that reduces cost can also become an attack surface for your agents.

Key takeaways

On 63,796 real queries, AWS measured an 86.3% cost cut and 88.3% lower end-to-end latency at a 0.75 cosine threshold, while answer accuracy held at 91.2% (AWS Database Blog, November 2025).
The similarity threshold is the whole control surface. Tightening it from 0.75 to 0.99 in the same tests dropped the cache hit rate from 90.3% to 23.5% and savings from 86.3% to 15.8%.
Once wrong-hit rates pass roughly 3%-5%, threshold tuning stops helping. That ceiling belongs to the embedding model, not the config (Portkey).
Semantic caches misfire in multi-turn dialogue. The ContextCache paper reports GPTCache-style lookups at 73.5% precision versus 94.3% once conversation context gates the match.
A 2026 study, CacheAttack, hijacked cached responses on AWS Bedrock (80.6%) and Azure (86.7%) semantic caches and cut agent answer accuracy by 83.8 percentage points. The defenses that work all reduce the hit rate.

What Is Semantic Caching, and How Does It Differ From Prompt and KV Caching?

Semantic caching stores past query-and-response pairs and serves a stored response when a new query means the same thing as an old one, judged by vector similarity rather than exact text. That "means the same thing" step is what separates it from the other two caches in a modern serving stack, and the distinction decides where each one saves money.

KV caching operates inside a single request, reusing the attention key and value tensors the model already computed for earlier tokens so decoding does not recompute them. Prompt caching operates across requests, storing the processed prefix of a prompt (a long system message, a document) so a provider skips re-encoding it. Semantic caching skips the model entirely: on a hit, only the embedding model runs, and the LLM is never called.

Cache type	What it reuses	Scope	Match rule	Wrong-answer risk
KV cache	Attention key/value tensors for tokens already decoded	Within one request (or a shared prefix)	Exact token prefix	None; it is a lossless recompute shortcut
Prompt cache	The encoded prefix of a prompt	Across requests sharing a prefix	Exact prefix bytes	None; the match is exact
Semantic cache	The full response to a prior query	Across all users and requests	Vector similarity above a threshold	Real; similar text can carry different intent

The order of magnitude differs too. KV and prompt caching trim the cost of a call you still make; semantic caching removes the call. That is why its savings run higher, and why its blast radius on a mistake runs higher as well. For the mechanics of the in-request layer, see why KV cache is the biggest lever in LLM inference cost.

How Much Does Semantic Caching Actually Save?

On a documented benchmark, up to 86% of inference cost. AWS engineers Meet Bhagdev, Chaitanya Nuthalapati, Jungwoo Song, and Utkarsh Shah ran 63,796 real chatbot queries from the SemBenchmarkLmArena dataset through Amazon ElastiCache for Valkey as a semantic cache in front of Amazon Bedrock, using Amazon Titan Text Embeddings V2 for 1024-dimensional vectors and Claude 3 Haiku as the model.

At a cosine similarity threshold of 0.75, they recorded a 90.3% cache hit ratio, 86.3% daily cost reduction, 88.3% lower end-to-end latency, and answer accuracy of 91.2%. Individual cache hits returned up to 59 times faster than a model call, because a hit invokes only the embedding model and a vector lookup.

Field reports track the lab numbers. A VentureBeat case study documented a team cutting monthly LLM spend from 47,000 dollars to 12,700 dollars, a 73% reduction, as the cache hit rate climbed from 18% to 67%. The savings scale with query repetition: support desks, documentation assistants, and internal knowledge bots see the highest hit rates because users ask the same few hundred questions in thousands of wordings. Token-heavy but low-repetition workloads see far less, which is why the decision belongs in the same budget conversation as token budgeting.

Why Does the Similarity Threshold Decide Everything?

Because it sets both the hit rate and the wrong-hit rate at the same time, and you cannot move one without moving the other. The threshold is the cosine similarity a new query must clear against a stored query to count as a match. Lower it and you catch more paraphrases; lower it too far and you catch different questions that happened to share vocabulary.

The AWS tests make the curve concrete.

Threshold	Cache hit rate	Answer accuracy	Cost savings
0.99 (strict)	23.5%	92.1%	15.8%
0.95	56.0%	92.6%	51.9%
0.90	74.5%	92.3%	72.5%
0.80	87.6%	91.8%	84.6%
0.75 (permissive)	90.3%	91.2%	86.3%

Read the accuracy column carefully. Across this range it barely moves, from 92.6% down to 91.2%, while savings swing from 15.8% to 86.3%. On a general chatbot corpus, the permissive threshold looks close to free. That is exactly the trap, because the corpus flatters it.

Portkey's guidance names the hard limit: once false positives exceed roughly 3%-5%, you have hit the embedding model's resolution, and no further threshold change will separate the intents it cannot tell apart. At that point the fix is a better embedding model, not a lower number. Their production default starts near 0.95 and backtests on about 5,000 queries until accuracy holds above 99%, rather than exposing the threshold as a user setting.

When Do Semantic Caches Return the Wrong Answer?

Most often in multi-turn conversation, where the same words mean different things depending on what came before. "What about the annual plan?" is a coherent cache key only if the cache knows the prior turn was about pricing for one product rather than another. Query-only similarity throws that context away.

The ContextCache paper quantifies the gap. On multi-turn sequences from ShareGPT, a GPTCache-style query-similarity cache reached 73.5% precision, meaning better than one in four hits carried a response that did not fit the conversation. Adding a context-consistency check on the dialogue history raised precision to 94.3% while holding a 67.2% hit rate and cutting cache-hit latency by 45%.

The lesson generalizes past that one system. A response is only reusable when the state that produced it still holds. Two failure classes recur: context drift, where the conversation has moved on, and entity ambiguity, where "reset it" or "the second one" resolves against different referents. A production cache needs its key to encode enough of that state, through conversation-aware keys, per-session namespaces, or metadata filters on user, tenant, and document. Domains where a wrong answer carries real cost, such as medical, legal, and financial assistants, warrant the strict thresholds of 0.95 and above that trade most of the savings for precision. This is the same reliability discipline covered in measuring AI agent reliability beyond the pass@1 number.

Can an Attacker Poison Your Semantic Cache?

Yes, and a 2026 study demonstrated it against commercial services. The paper, "From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Caching," introduces CacheAttack, which treats a semantic cache key as a locality-preserving fuzzy hash and crafts adversarial suffixes that collide with a victim query's key. On a collision, the system serves the attacker's cached response in place of the model's.

The measured success rates are not marginal. CacheAttack variants hijacked responses with 86.9% and 83.1% hit rates, and injected attacker content with 81.1% and 77.1% success. Treating the embedding model as a black box and using public surrogates, the attack still worked on AWS Bedrock (80.6%) and Azure (86.7%). Against an agent, a poisoned cache hit landed 93.2% of the time and dropped answer accuracy by 83.8 percentage points by steering tool selection. Attacks transferred across embedding models at 49%-94% hit rates, so a secret in-house model is not by itself a defense.

The proposed defenses each cost something. Key salting, mixing a per-deployment secret into the cache key, cut attack success by 9 to 25 percentage points. Perplexity screening rejects unnatural inputs at insertion time. Per-user cache isolation eliminates cross-user poisoning outright but also erases the cross-user hits that make the cache pay. The paper is blunt that every one of these lowers the hit rate, which reframes semantic caching as a security-sensitive component, not a passive optimization. Anyone running agents should read it next to how to securely isolate AI agents with scoped identities.

Where Should Semantic Caching Sit in Your Inference Stack?

As the outermost layer, in front of prompt caching, KV caching, and model routing, because it is the only one that can avoid the model call altogether. Practitioner reports converge on a stack: semantic caching plus prompt caching plus model routing delivers 47%-80% cost reduction, and teams that run all of them reach 70%-90% in production.

Ordering matters. Check the semantic cache first; on a miss, route the request to the cheapest model that clears the task, then let prompt and KV caching cut the cost of the call you could not avoid. One more layer is worth adding for high-stakes answers: verification before storage. AWS publishes a sample, the Verified Semantic Cache, that grounds a candidate answer against a Bedrock knowledge base before caching it, so the cache cannot amplify a hallucination across every future hit. That converts the cache from a pure speed play into a correctness gate, at the cost of a verification step on writes.

The RevOps read is straightforward. Semantic caching moves inference cost from a per-query variable into a function of query diversity, which finance can forecast. But it hands an operational lever to whoever owns the threshold, and that person is now making an accuracy and a security decision, not a caching decision.

What Are the Trade-offs to Watch?

The core tension does not resolve; you position on it. Every gain in hit rate is a step toward serving a response that no longer fits the question, whether through paraphrase collision, context drift, or a crafted adversarial suffix. The benchmark accuracy numbers, measured on general chatbot corpora, will overstate your safety on a narrow high-stakes domain where near-duplicate questions demand different answers.

Three things deserve monitoring in production. Track the wrong-hit rate directly with periodic sampled human or model review, not just the hit rate, because the hit rate looks better exactly as precision degrades. Watch the false-positive ceiling: if sampled precision stalls near 95%-97% no matter how you tune, the embedding model has topped out and needs replacing. And treat cache writes as a trust boundary, since anything an attacker can get stored, every later user can be served.

The teams that get this right stop treating semantic caching as free savings and start treating it as a precision budget they spend deliberately, with the threshold, the key design, and the write path all under review.