Skip to content
BENIBIX Learn

Model Memory Requirements

AI💻 Running LocallyLocal Models

Also known as VRAM Requirements, LLM Memory Requirements, GPU Memory Requirements

How much memory a model needs to run. Weights first, then room for the chat.

Think of it like

It is like packing a suitcase. It all has to fit, or the lid will not shut.

Example

Emma's graphics card has 8 GB of memory. An 8 billion weight model at 16 bits needs about 16 GB, so it will not fit. The same model shrunk to Q4_K_M is about 5 GB, and it runs fast.

Why it matters

It tells you before you download if a model will run on your machine, and how fast.

Definition

Model memory requirements are how much memory a model needs to run. The biggest part is the weights: the count of weights times the bytes each one takes. An 8 billion weight model at 16 bits needs about 16 GB just for that. Shrunk to about 5 bits a weight, it needs about 5 GB. On top of that comes room for the chat so far, which grows as the chat gets longer. If it does not all fit in the graphics card's memory, part spills into slower memory and it slows down a lot.
Weights, then room for the chat, then the program. All of it under the line runs fast.

Common mistake

Counting only the file size. A long chat needs more room on top, so leave some memory spare.

Key takeaways

  1. 1
    Weights first. Count times the bytes for each.
  2. 2
    The chat needs room. Long chats take more memory.
  3. 3
    Fit or crawl. Spilling over slows it down a lot.