Skip to content
BENIBIX Learn

Quantization (AI)

AI💻 Running LocallyModel Compression

Also known as Model Quantization, LLM Quantization, 4-bit Quantization

A way to shrink a model. Each weight is kept with fewer bits.

Think of it like

Think of it like rounding prices to the nearest dollar. The list is shorter, and you lose just a bit.

Example

Lucy's laptop has 16 GB of memory. The model she wants is 16 GB at full size, so she downloads a 4-bit version of about 5 GB, and it runs well.

Why it matters

It is the main reason good models fit on normal laptops and phones at all.

Definition

Quantization makes a model smaller by storing each weight with fewer bits. Weights are usually trained as 16-bit numbers, and quantization rounds them to 8 bits, 4 bits or even less. An 8 billion weight model drops from about 16 GB at 16 bits to about 5 GB at 4 bits. The model also runs faster, because there is less to read from memory for each token. At 8 or 4 bits the loss in quality is usually small, but it grows quickly below that.
The same weight with 16, 8 and 4 bits. The file shrinks, and the number gets a little rougher.

Common mistake

Going as low as possible to save space. Below about 4 bits, answers often get noticeably worse, so test before you settle.

Key takeaways

  1. 1
    Fewer bits per weight. 16 down to 4.
  2. 2
    Smaller and faster. Less to read.
  3. 3
    Small loss at first. Big loss if pushed.

Compare