Anyone who has waited for one slow web page knows the feeling: the site is usually fast, and then, once in a while, it just stalls. For a small site this is an annoyance. For a service like Google Search it is a design problem, because one page view quietly asks hundreds of machines for help.
Jeff Dean and Luiz Barroso's 2013 article explains why the rare slow response, the tail of the latency distribution, dominates what users feel at scale. It also gives a toolbox of cheap tricks that tolerate slowness instead of trying to remove every cause.
If a request waits for many servers, even a 1-in-100 slowdown per server hits most requests. You cannot remove all the causes of slowness, so design the system to tolerate it: send backup requests, cancel duplicates, and keep slow machines out of the path.
Why rare delays become common
Take the paper's example. A server answers in 10 milliseconds most of the time, but one request in a hundred takes a full second. If a user request touches just one such server, one user in a hundred waits a second. That sounds tolerable.
Now suppose a request must collect answers from 100 of those servers in parallel. It finishes only when the slowest answer arrives. The paper computes that 63 percent of user requests will now take more than one second.
It is like a group dinner reservation where the table is only ready when the last guest arrives. Even if each person is late only 1 time in 100, a party of 100 is almost never complete on time.
The math is simple. If each server is slow with probability p, a request waiting on N servers is slow with probability 1 minus (1 minus p) to the power N. Play with it below.
