# KV Cache

> Saved work for the tokens so far, so each new token is quick.

**Field:** Artificial Intelligence  
**Category:** LLMs  
**Also known as:** Key-Value Cache, Attention Cache

## Think of it like

Think of it like notes you keep as you read. Then you do not have to read the page again.

## Definition

The KV cache stores the keys and values that attention works out for each token already in the text. When the model writes the next token, it reads these from the cache. It does not work them out again. That makes each new token much faster to make. The cost is memory: the cache grows with every token, and long chats can fill gigabytes.

## Example

Lucy's model writes a 2,000-token answer. With a KV cache, each new token reuses the work for all the ones before instead of redoing it.

## Why it matters

It is why long answers are cheap enough to make. It is also why long chats use so much memory.

## Common mistake

Planning memory for the model's weights alone. A long chat can need many more gigabytes for the cache.

## Three principles

- **Saves keys and values.** For tokens already seen.
- **Reuses work.** Each new token is fast.
- **Costs memory.** Grows with every token.

## Related terms

- [Self-Attention](https://learn.benibix.com/term/ai/self-attention)
- [Context Window](https://learn.benibix.com/term/ai/context-window)
- [Video Memory (VRAM)](https://learn.benibix.com/term/infrastructure/video-memory)
- [Prompt Caching](https://learn.benibix.com/term/ai/prompt-caching)
- [vLLM](https://learn.benibix.com/term/ai/vllm)
- [Speculative Decoding](https://learn.benibix.com/term/ai/speculative-decoding)

## Taught in

- [LLM Speed and Streaming](https://learn.benibix.com/learn/ai-llms/llm-speed-and-streaming): LLMs, 13/15

![Earlier tokens have their keys and values stored in a cache. The new token works out only its own and reads the rest from the cache.](https://learn.benibix.com/diagrams/llm-kv-cache.svg)

---

Source: https://learn.benibix.com/term/ai/kv-cache
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
