Attention is the part of a transformer that lets each word look at every other word. It is powerful, and it gets expensive quickly: for a text of N tokens, attention builds an N by N table of scores. Double the text and the table grows four times. For years, researchers tried to make attention cheaper by approximating it, which usually cost some accuracy and, surprisingly, often did not even make it faster in practice.
The FlashAttention paper from Stanford and the University at Buffalo found that the real bottleneck was not arithmetic. It was memory traffic: how often data moves between a GPU's slow big memory and its fast tiny memory. By reorganizing the computation to move less data, they made exact attention several times faster and far more memory-efficient.
FlashAttention computes exactly the same result as standard attention, but processes it in small tiles that stay in the GPU's fast memory, so the huge N by N matrix is never written to slow memory. It is faster and uses memory that grows only linearly with sequence length.
The real cost: moving data
A modern GPU has several kinds of memory. The paper's figure lists an A100-class GPU. SRAM, on the chip, is very fast (about 19 terabytes per second) but holds only about 20 megabytes. HBM, the GPU's main memory, holds around 40 gigabytes but moves data at about 1.5 terabytes per second, an order of magnitude slower. CPU memory is slower still.
As GPUs have become faster at arithmetic than at fetching data, many operations are limited by memory access. The paper calls an algorithm IO-aware if it carefully counts reads and writes between these levels, not only the arithmetic.
Think of a cook with a tiny countertop and a big pantry down the hall. The speed of the meal depends mostly on how many trips to the pantry you make, not how fast you chop. Standard attention makes a trip for every intermediate result. FlashAttention carries a small batch, cooks it completely, and returns once.
