# FlashAttention

> A faster way to work out attention on a GPU. The answer stays exact.

**Field:** Artificial Intelligence  
**Category:** LLMs  
**Also known as:** Flash Attention, FlashAttention-2

## Think of it like

It is like doing sums at a small desk. You fetch a few pages at a time, not the whole heavy book.

## Definition

FlashAttention is a faster way to work out attention on a GPU, with the exact same result. Plain attention builds a huge table of scores. It keeps moving that table in and out of the GPU's big, slow memory. FlashAttention cuts the work into small tiles. Each tile fits in the chip's small, fast memory, so the whole table is never stored. The 2022 paper that brought it in showed GPT-2 training about 3 times faster.

## Example

A team trains a model on long reports. With FlashAttention turned on, the same GPUs take longer texts without running out of memory.

## Why it matters

Attention is the slowest, hungriest part of a long text. This saves time and memory without changing the answer.

## Common mistake

Thinking it guesses or changes the model. It works out the same attention, only in a smarter order.

## Three principles

- **Exact result.** No shortcut in the math.
- **Small tiles.** They fit in fast memory.
- **Less memory.** The big table is never stored.

## Related terms

- [Attention Mechanism](https://learn.benibix.com/term/ai/attention-mechanism)
- [Self-Attention](https://learn.benibix.com/term/ai/self-attention)
- [Multi-Head Attention](https://learn.benibix.com/term/ai/multi-head-attention)
- [Context Window](https://learn.benibix.com/term/ai/context-window)
- [Graphics Processing Unit (GPU)](https://learn.benibix.com/term/infrastructure/graphics-processing-unit)
- [Video Memory (VRAM)](https://learn.benibix.com/term/infrastructure/video-memory)

## Taught in

- [Self-Attention and Multi-Head](https://learn.benibix.com/learn/ai-llms/self-attention-and-multi-head): LLMs, 6/15

![Left: the full table of attention scores is written out to the GPU's big, slow memory and read back. Right: the table is split into small tiles, and each tile is worked out inside the small, fast memory on the chip.](https://learn.benibix.com/diagrams/ai9-flash-attention.svg)

---

Source: https://learn.benibix.com/term/ai/flash-attention
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
