Skip to content
Metastable Failures in Distributed Systems 4 min

Nathan Bronson, Abutalib Aghayev, Aleksey Charapko, Timothy Zhu · HotOS 2021

Metastable Failures in Distributed Systems

Why some outages outlast their cause: a feedback loop such as retries keeps the system down. Trigger an outage and compare retry policies.

Read the original paper

Systems & Distributed · Yellow Belt · Pro · 4 min read

Metastable Failures in Distributed Systems

Some outages are caused by something that broke. A disk fails, a bug ships, a network cable is cut. You fix the cause and the system comes back. But some outages keep going after the cause is gone. The system looks healthy in every way except that it does not work, and it stays that way for minutes or hours.

A 2021 workshop paper from engineers and researchers who had run large systems for years names this pattern metastable failure. It describes how to recognize it, shows common examples, and surveys the fixes people had invented after the fact. It is one of the clearest explanations of why well-meant features, such as retries and caches, can turn a hiccup into an outage.

A metastable failure has three parts. A system runs in a vulnerable state, a trigger such as a brief outage pushes it over, and then a feedback loop, the sustaining effect, keeps it down even after the trigger has gone. The usual culprits are features that help in good times, such as retries and caches, which add work when the system is already struggling.

Why the trigger is not the cause

The paper defines the pattern for open systems, where load comes from an uncontrolled source such as users. A trigger makes the system enter a bad state that persists even when the trigger is removed. In that state the goodput, the rate of useful work, is very low.

The life cycle in the paper's Figure 1. A system runs fine while load is low. In a vulnerable state, a trigger can push it into a bad state that a feedback loop then sustains, even after the trigger is removed.

The authors stress that the trigger is not the root cause. Many things can trigger it, such as a load spike, a brief network failure or a software bug. What matters is the sustaining effect, a loop that feeds on itself. Fixing the trigger does not help. You must break the loop.

Another uncomfortable point: many production systems run in the vulnerable state all the time, because that is where they are efficient. They can run there for months, or years, before a trigger finds them.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 2 interactive elements to play with

  • The retry example
  • Other sustaining effects, and what helps

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.