💬KV Cache
Also known as Key-Value Cache, Attention Cache
Saved work for the tokens so far, so each new token is quick.
Think of it like
Think of it like notes you keep as you read. Then you do not have to read the page again.
Example
Lucy's model writes a 2,000-token answer. With a KV cache, each new token reuses the work for all the ones before instead of redoing it.
Why it matters
It is why long answers are cheap enough to make. It is also why long chats use so much memory.
Definition
Common mistake
Key takeaways
- 1Saves keys and values. For tokens already seen.
- 2Reuses work. Each new token is fast.
- 3Costs memory. Grows with every token.