Caching Strategies for LLM Responses
You'll learn to
- -Understand exact-match vs. semantic caching for LLM calls
- -Reason about cache invalidation for generated content
- -See exactly how a similarity threshold decides cache hits, with real cosine similarity
Forty percent of a company's LLM queries turn out to be semantically identical to a query someone already asked, just worded differently: "what is the return policy" and "can I return this item" are the same question in different clothes. Every one of those still costs a full LLM call under a naive setup. Caching is the single highest-leverage lever for cutting LLM spend, and it is also one of the easiest ones to get subtly wrong.
Two Kinds of Cache
- -Exact-match caching keys on the literal input string. Fast and completely safe, but only catches queries that are character-for-character identical, which is a small fraction of real duplicate traffic.
- -Semantic caching embeds the incoming query, from the Tier 7 embeddings chapter, and checks its similarity against previously cached queries. This catches "what is the return policy" and "can I return this item" as the same cache entry, at the cost of introducing a genuine risk: a wrong match returns a wrong cached answer.
That risk is exactly why the threshold matters so much. Drag it below and watch the same three cached queries flip between hit and miss.
Where You Set the Threshold Decides the Answer
New query: "What is our return policy for electronics?". Drag the threshold and watch which cached queries count as a match, computed from real cosine similarity.
Cache hit, return instantly
The closest cached entry (sim 0.9993) clears the 0.93 threshold, so this query gets the cached answer for free.
A threshold that is too low serves confidently wrong answers to genuinely different questions that merely resemble each other in embedding space. A threshold that is too high defeats the entire purpose of semantic caching, catching almost nothing beyond exact duplicates. The case brief's own guidance lands around 0.93 to 0.97 for most domains, and the right number ultimately depends on how costly a wrong cached answer is for your specific use case.
Cache Invalidation for Generated Content
A cached answer that was correct yesterday can be wrong today: a return policy changes, a price updates, an inventory count shifts. Every cache entry needs a time-to-live, commonly 24 hours or less for anything that can go stale, after which it expires and the next matching query pays for a fresh LLM call. This is the same staleness problem every cache has always had, just applied to generated text instead of a database row.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Build the Serving Engine, Gateway, and Cache Layer levels in the GenAI Lab.