Skip to content
BENIBIX Learn

KV Cache

AI💬 LLMsText Generation

Also known as Key-Value Cache, Attention Cache

Saved work for the tokens so far, so each new token is quick.

Think of it like

Think of it like notes you keep as you read. Then you do not have to read the page again.

Example

Lucy's model writes a 2,000-token answer. With a KV cache, each new token reuses the work for all the ones before instead of redoing it.

Why it matters

It is why long answers are cheap enough to make. It is also why long chats use so much memory.

Definition

The KV cache stores the keys and values that attention works out for each token already in the text. When the model writes the next token, it reads these from the cache. It does not work them out again. That makes each new token much faster to make. The cost is memory: the cache grows with every token, and long chats can fill gigabytes.
The new token works out only its own part and reads the rest from the cache.

Common mistake

Planning memory for the model's weights alone. A long chat can need many more gigabytes for the cache.

Key takeaways

  1. 1
    Saves keys and values. For tokens already seen.
  2. 2
    Reuses work. Each new token is fast.
  3. 3
    Costs memory. Grows with every token.

Learn it properly