Semantic Cache vs Prompt Caching
People often think a semantic cache and prompt caching are the same money saving trick. A semantic cache skips the model and returns a saved answer. It does that when a new question means the same as an old one. Prompt caching still calls the model every time, but it reuses the work done on the start of a long prompt that has not changed.
Semantic Cache
A store that gives back a saved answer when a new question means the same.
A semantic cache stores model answers and reuses them for new questions that mean the same thing. Each question is turned into an embedding, a list of numbers that captures its meaning. When a new question comes in, the cache looks for a saved one that is close enough. If it finds one, it returns the saved answer at once, with no model call. If not, the model answers and the new pair is saved.
Prompt Caching
Saving the work on a prompt's fixed start, so the next request reuses it.
Prompt caching saves the model's work on the start of a prompt, so the next request that starts the same way can reuse it. It helps when many requests share a long fixed part, like a system prompt, a manual or a codebase. The reused part is read much faster and billed at a lower price. The cache only lasts a short time, often minutes, and the start has to match exactly. Any change early in the prompt breaks the match for everything after it.
In one line
- Semantic Cache
- A store that gives back a saved answer when a new question means the same.
- Prompt Caching
- Saving the work on a prompt's fixed start, so the next request reuses it.
These live in different corners of the glossary (Development and AI), which is exactly why they get confused.