Skip to content
BENIBIX Learn

Quantization vs Model Pruning

People often think quantization and pruning are the same way to shrink a model. Quantization keeps every weight but saves each one with fewer bits, like rounding. Pruning takes out the weights or parts that matter least, so there are fewer of them. The two work in their own ways, so they are often used together.

AI💻 Running Locally

Quantization (AI)

A way to shrink a model. Each weight is kept with fewer bits.

Quantization makes a model smaller by storing each weight with fewer bits. Weights are usually trained as 16-bit numbers, and quantization rounds them to 8 bits, 4 bits or even less. An 8 billion weight model drops from about 16 GB at 16 bits to about 5 GB at 4 bits. The model also runs faster, because there is less to read from memory for each token. At 8 or 4 bits the loss in quality is usually small, but it grows quickly below that.

Example
Lucy's laptop has 16 GB of memory. The model she wants is 16 GB at full size, so she downloads a 4-bit version of about 5 GB, and it runs well.
Why it matters
It is the main reason good models fit on normal laptops and phones at all.
Full entry
AI💻 Running Locally

Model Pruning

A way to shrink a model. The parts it needs least are cut out.

Model pruning makes a model smaller by taking out the parts that matter least. It can take out single weights that are close to zero. Or it can take out whole parts, such as neurons, attention heads or layers. After pruning, the model is often trained a bit more, so it can make up for what was lost. Taking out whole parts is what makes it run faster on normal hardware. Pruning is often used with quantization and distillation.

Example
A team prunes a quarter of the layers from a speech model and trains it for one more day. It runs a third faster on phones and scores almost the same.
Why it matters
It shrinks a model in a new way, not like quantization. So the two can be used at once.
Full entry

In one line

Quantization (AI)
A way to shrink a model. Each weight is kept with fewer bits.
Model Pruning
A way to shrink a model. The parts it needs least are cut out.