💻Model Memory Requirements
Also known as VRAM Requirements, LLM Memory Requirements, GPU Memory Requirements
How much memory a model needs to run. Weights first, then room for the chat.
Think of it like
It is like packing a suitcase. It all has to fit, or the lid will not shut.
Example
Emma's graphics card has 8 GB of memory. An 8 billion weight model at 16 bits needs about 16 GB, so it will not fit. The same model shrunk to Q4_K_M is about 5 GB, and it runs fast.
Why it matters
It tells you before you download if a model will run on your machine, and how fast.
Definition
Common mistake
Key takeaways
- 1Weights first. Count times the bytes for each.
- 2The chat needs room. Long chats take more memory.
- 3Fit or crawl. Spilling over slows it down a lot.