Memory Bandwidth vs Network Bandwidth
People often hear bandwidth and think of internet speed. Network bandwidth is how much data moves over a link between machines. Memory bandwidth is how much moves inside one machine, between memory and a chip. The second is often thousands of times bigger. It is the one that sets how fast AI writes.
Memory Bandwidth
How fast data moves from memory to the chip. It often sets AI speed.
Memory bandwidth is how much data can move between memory and a chip each second. It is measured in gigabytes per second. A chip that does sums very fast still has to wait if the data comes in slowly. When an AI model writes text, it reads all of its weights for every word, so this speed often sets how fast the words come out. GPU memory is many times faster than normal RAM.
Bandwidth
The most data a line can carry in one second.
Bandwidth is the most data a network link can carry each second. It is usually counted in megabits per second, written Mbps, or gigabits per second. A 100 Mbps line can, at best, move about 12.5 megabytes each second, since a byte is 8 bits. It is a limit, not a promise: the real rate is often lower. Hosts also use the word for how much data a server may send in a month.
In one line
- Memory Bandwidth
- How fast data moves from memory to the chip. It often sets AI speed.
- Bandwidth
- The most data a line can carry in one second.