💬FlashAttention
Also known as Flash Attention, FlashAttention-2
A faster way to work out attention on a GPU. The answer stays exact.
Think of it like
It is like doing sums at a small desk. You fetch a few pages at a time, not the whole heavy book.
Example
A team trains a model on long reports. With FlashAttention turned on, the same GPUs take longer texts without running out of memory.
Why it matters
Attention is the slowest, hungriest part of a long text. This saves time and memory without changing the answer.
Definition
Common mistake
Key takeaways
- 1Exact result. No shortcut in the math.
- 2Small tiles. They fit in fast memory.
- 3Less memory. The big table is never stored.