Skip to content
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness 4 min

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré · NeurIPS 2022

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Exact attention that is several times faster by moving less data: tiling, online softmax and the GPU memory hierarchy, with two calculators.

Read the original paper

AI & ML · Black Belt · Pro · 4 min read

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

Attention is the part of a transformer that lets each word look at every other word. It is powerful, and it gets expensive quickly: for a text of N tokens, attention builds an N by N table of scores. Double the text and the table grows four times. For years, researchers tried to make attention cheaper by approximating it, which usually cost some accuracy and, surprisingly, often did not even make it faster in practice.

The FlashAttention paper from Stanford and the University at Buffalo found that the real bottleneck was not arithmetic. It was memory traffic: how often data moves between a GPU's slow big memory and its fast tiny memory. By reorganizing the computation to move less data, they made exact attention several times faster and far more memory-efficient.

FlashAttention computes exactly the same result as standard attention, but processes it in small tiles that stay in the GPU's fast memory, so the huge N by N matrix is never written to slow memory. It is faster and uses memory that grows only linearly with sequence length.

The real cost: moving data

A modern GPU has several kinds of memory. The paper's figure lists an A100-class GPU. SRAM, on the chip, is very fast (about 19 terabytes per second) but holds only about 20 megabytes. HBM, the GPU's main memory, holds around 40 gigabytes but moves data at about 1.5 terabytes per second, an order of magnitude slower. CPU memory is slower still.

Numbers from the paper's Figure 1 for an A100-class GPU. On-chip SRAM is much faster than GPU memory (HBM), but tiny. The speed of attention depends heavily on how often data must travel between them.

As GPUs have become faster at arithmetic than at fetching data, many operations are limited by memory access. The paper calls an algorithm IO-aware if it carefully counts reads and writes between these levels, not only the arithmetic.

Think of a cook with a tiny countertop and a big pantry down the hall. The speed of the meal depends mostly on how many trips to the pantry you make, not how fast you chop. Standard attention makes a trip for every intermediate result. FlashAttention carries a small batch, cooks it completely, and returns once.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

3 more sections, 1 more diagram and 3 interactive elements to play with

  • What standard attention wastes
  • The trick that makes tiling possible: online softmax
  • Results and what it changed

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.