Skip to content
BENIBIX Learn

Semantic Cache

Development🧬 AI EngineeringLLM App Operations

Also known as Semantic Caching, LLM Cache

A store that gives back a saved answer when a new question means the same.

Think of it like

Think of it like a teacher who hears the same question in new words. They give the same answer again.

Example

Lucy's help bot gets "how do I reset my password" in dozens of ways. The semantic cache answers most of them in a few milliseconds. It cuts her bill by a third.

Why it matters

Repeat questions are common, and a cache hit is both much faster and almost free.

Definition

A semantic cache stores model answers and reuses them for new questions that mean the same thing. Each question is turned into an embedding, a list of numbers that captures its meaning. When a new question comes in, the cache looks for a saved one that is close enough. If it finds one, it returns the saved answer at once, with no model call. If not, the model answers and the new pair is saved.
A close match returns the saved answer at once. No match goes on to the model.

Common mistake

Setting the match too loose. Two questions can be close in meaning but need different answers, and a loose cache serves the wrong one.

Key takeaways

  1. 1
    Match by meaning. Not exact words.
  2. 2
    A hit skips the model. Fast and cheap.
  3. 3
    Tune the threshold. Too loose is wrong.

Compare