Skip to content
BENIBIX Learn

Model Precision

AI💻 Running LocallyModel Compression

Also known as FP16, F16, BF16, Bfloat16, FP32, Half Precision, Floating-Point Precision

How many bits each weight in a model gets. Fewer bits make a smaller model.

Think of it like

It is like writing a price as 3.14159 or just 3.14. The short one takes less room.

Example

A model page lists the weights as BF16. Jack knows that means 2 bytes a weight. So a 7 billion weight model is about 14 GB.

Why it matters

It sets the file size. It also sets the memory a model needs before any shrinking.

Definition

Model precision is how many bits each weight in a model gets. Full precision is 32 bits a weight, called FP32. Most open models are shared at 16 bits, as FP16 or BF16, which is half the size. BF16 holds the same range of big and small numbers as FP32, with less detail in each. Fewer bits mean a smaller file and faster running. Going below 16 bits is called quantization.
Full precision and 16 bits sit at the top. Each step down keeps fewer bits a weight.

Common mistake

Thinking FP16 and BF16 are the same. Both use 16 bits, but BF16 can hold far bigger and smaller numbers, with less detail in each.

Key takeaways

  1. 1
    Bits per weight. Size grows with the bits.
  2. 2
    16 is normal. Half the size of FP32.
  3. 3
    Below 16. That step is called quantization.