Skip to content
BENIBIX Learn

FlashAttention

AI💬 LLMsTransformer Architecture

Also known as Flash Attention, FlashAttention-2

A faster way to work out attention on a GPU. The answer stays exact.

Think of it like

It is like doing sums at a small desk. You fetch a few pages at a time, not the whole heavy book.

Example

A team trains a model on long reports. With FlashAttention turned on, the same GPUs take longer texts without running out of memory.

Why it matters

Attention is the slowest, hungriest part of a long text. This saves time and memory without changing the answer.

Definition

FlashAttention is a faster way to work out attention on a GPU, with the exact same result. Plain attention builds a huge table of scores. It keeps moving that table in and out of the GPU's big, slow memory. FlashAttention cuts the work into small tiles. Each tile fits in the chip's small, fast memory, so the whole table is never stored. The 2022 paper that brought it in showed GPT-2 training about 3 times faster.
The old way sends the whole table to slow memory. This way does one small tile at a time in fast memory.

Common mistake

Thinking it guesses or changes the model. It works out the same attention, only in a smarter order.

Key takeaways

  1. 1
    Exact result. No shortcut in the math.
  2. 2
    Small tiles. They fit in fast memory.
  3. 3
    Less memory. The big table is never stored.

Learn it properly