Skip to content
The Tail at Scale 4 min

Jeffrey Dean, Luiz André Barroso · Communications of the ACM 2013

The Tail at Scale

Why the rare slow response ruins big systems, and how hedged requests, tied requests and good-enough answers keep the tail short. Includes a fan-out calculator.

Read the original paper

Systems & Distributed · Yellow Belt · Pro · 4 min read

The Tail at Scale

Anyone who has waited for one slow web page knows the feeling: the site is usually fast, and then, once in a while, it just stalls. For a small site this is an annoyance. For a service like Google Search it is a design problem, because one page view quietly asks hundreds of machines for help.

Jeff Dean and Luiz Barroso's 2013 article explains why the rare slow response, the tail of the latency distribution, dominates what users feel at scale. It also gives a toolbox of cheap tricks that tolerate slowness instead of trying to remove every cause.

If a request waits for many servers, even a 1-in-100 slowdown per server hits most requests. You cannot remove all the causes of slowness, so design the system to tolerate it: send backup requests, cancel duplicates, and keep slow machines out of the path.

Why rare delays become common

Take the paper's example. A server answers in 10 milliseconds most of the time, but one request in a hundred takes a full second. If a user request touches just one such server, one user in a hundred waits a second. That sounds tolerable.

Now suppose a request must collect answers from 100 of those servers in parallel. It finishes only when the slowest answer arrives. The paper computes that 63 percent of user requests will now take more than one second.

A search query fans out to many servers and cannot finish until the last one answers. A rare slowdown on any single server becomes a common slowdown for the user.

It is like a group dinner reservation where the table is only ready when the last guest arrives. Even if each person is late only 1 time in 100, a party of 100 is almost never complete on time.

The math is simple. If each server is slow with probability p, a request waiting on N servers is slow with probability 1 minus (1 minus p) to the power N. Play with it below.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 3 interactive elements to play with

  • Why servers are sometimes slow
  • Cheap tricks that tame the tail

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.