💻CPU Offloading
Also known as Partial GPU Offload, GPU Layer Offloading, Layer Offloading
If a model is too big for the GPU, part of it runs on the main chip.
Think of it like
Think of it like a desk too small for all your books. Some go on the floor, and you reach down for them.
Example
Jack's graphics card has 8 GB of memory and his model needs 12 GB. He puts 24 of its layers on the card and the rest on the CPU, and it runs, just slower.
Why it matters
It lets a model run on hardware that is a little too small, instead of not at all.
Definition
Example code
bash
llama-cli -m model.gguf --n-gpu-layers 24