Skip to content
BENIBIX Learn

CPU Offloading

AI💻 Running LocallyLocal Models

Also known as Partial GPU Offload, GPU Layer Offloading, Layer Offloading

If a model is too big for the GPU, part of it runs on the main chip.

Think of it like

Think of it like a desk too small for all your books. Some go on the floor, and you reach down for them.

Example

Jack's graphics card has 8 GB of memory and his model needs 12 GB. He puts 24 of its layers on the card and the rest on the CPU, and it runs, just slower.

Why it matters

It lets a model run on hardware that is a little too small, instead of not at all.

Definition

CPU offloading runs part of a model on the main chip when the GPU is too small. The model's layers are split. As many as fit go on the graphics card, and the rest stay in main memory and run on the CPU. This lets a model run that would not fit at all. The cost is speed, since the CPU part is much slower. Many local tools let you choose how many layers go on the card.

Example code

bash
llama-cli -m model.gguf --n-gpu-layers 24
Put 24 layers on the graphics card. The rest run on the CPU.
Most layers run fast on the graphics card. The rest run slowly on the CPU.

Common mistake

Offloading most of the model and expecting graphics card speed. When most layers are on the CPU, it can be almost as slow as no graphics card.

Key takeaways

  1. 1
    Split the layers. Card and CPU.
  2. 2
    Fits, but slower. The CPU part drags.
  3. 3
    Choose the split. Layers on the card.