# CPU Offloading

> If a model is too big for the GPU, part of it runs on the main chip.

**Field:** Artificial Intelligence  
**Category:** Running Locally  
**Also known as:** Partial GPU Offload, GPU Layer Offloading, Layer Offloading

## Think of it like

Think of it like a desk too small for all your books. Some go on the floor, and you reach down for them.

## Definition

CPU offloading runs part of a model on the main chip when the GPU is too small. The model's layers are split. As many as fit go on the graphics card, and the rest stay in main memory and run on the CPU. This lets a model run that would not fit at all. The cost is speed, since the CPU part is much slower. Many local tools let you choose how many layers go on the card.

## Example

Jack's graphics card has 8 GB of memory and his model needs 12 GB. He puts 24 of its layers on the card and the rest on the CPU, and it runs, just slower.

## Example code

```bash
llama-cli -m model.gguf --n-gpu-layers 24
```

Put 24 layers on the graphics card. The rest run on the CPU.

## Why it matters

It lets a model run on hardware that is a little too small, instead of not at all.

## Common mistake

Offloading most of the model and expecting graphics card speed. When most layers are on the CPU, it can be almost as slow as no graphics card.

## Three principles

- **Split the layers.** Card and CPU.
- **Fits, but slower.** The CPU part drags.
- **Choose the split.** Layers on the card.

## Related terms

- [Video Memory (VRAM)](https://learn.benibix.com/term/infrastructure/video-memory)
- [Central Processing Unit (CPU)](https://learn.benibix.com/term/infrastructure/central-processing-unit)
- [Random Access Memory (RAM)](https://learn.benibix.com/term/infrastructure/random-access-memory)
- [Quantization (AI)](https://learn.benibix.com/term/ai/quantization)
- [Tokens per Second](https://learn.benibix.com/term/ai/tokens-per-second)
- [Model Memory Requirements](https://learn.benibix.com/term/ai/model-memory-requirements)

![A model's layers split in two. Most fit in the graphics card's memory and run fast. The rest sit in main memory and run on the CPU, which is slower.](https://learn.benibix.com/diagrams/lo-cpu-offloading.svg)

---

Source: https://learn.benibix.com/term/ai/cpu-offloading
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
