Skip to content
BENIBIX Learn

Memory Bandwidth

Infrastructure🔧 HardwareMemory

Also known as Memory Throughput

How fast data moves from memory to the chip. It often sets AI speed.

Think of it like

Think of it like the width of a road to a shop. A wide road brings more goods each minute.

Example

Lucy's GPU moves about 1,000 GB per second. Her model is 10 GB, so the GPU can read it about 100 times a second. That caps it at about 100 words a second.

Why it matters

It explains why two chips with the same raw speed can run AI at very different rates.

Definition

Memory bandwidth is how much data can move between memory and a chip each second. It is measured in gigabytes per second. A chip that does sums very fast still has to wait if the data comes in slowly. When an AI model writes text, it reads all of its weights for every word, so this speed often sets how fast the words come out. GPU memory is many times faster than normal RAM.
The width of each pipe is the memory bandwidth. The GPU's pipe is many times wider.

Common mistake

Judging an AI chip only by how many sums it can do. If memory cannot feed it, those sums sit idle.

Key takeaways

  1. 1
    Data per second. From memory to chip.
  2. 2
    Can be the limit. Fast chips still wait.
  3. 3
    Sets text speed. Weights are read for every word.

Compare

Learn it properly