Some outages are caused by something that broke. A disk fails, a bug ships, a network cable is cut. You fix the cause and the system comes back. But some outages keep going after the cause is gone. The system looks healthy in every way except that it does not work, and it stays that way for minutes or hours.
A 2021 workshop paper from engineers and researchers who had run large systems for years names this pattern metastable failure. It describes how to recognize it, shows common examples, and surveys the fixes people had invented after the fact. It is one of the clearest explanations of why well-meant features, such as retries and caches, can turn a hiccup into an outage.
A metastable failure has three parts. A system runs in a vulnerable state, a trigger such as a brief outage pushes it over, and then a feedback loop, the sustaining effect, keeps it down even after the trigger has gone. The usual culprits are features that help in good times, such as retries and caches, which add work when the system is already struggling.
Why the trigger is not the cause
The paper defines the pattern for open systems, where load comes from an uncontrolled source such as users. A trigger makes the system enter a bad state that persists even when the trigger is removed. In that state the goodput, the rate of useful work, is very low.
The authors stress that the trigger is not the root cause. Many things can trigger it, such as a load spike, a brief network failure or a software bug. What matters is the sustaining effect, a loop that feeds on itself. Fixing the trigger does not help. You must break the loop.
Another uncomfortable point: many production systems run in the vulnerable state all the time, because that is where they are efficient. They can run there for months, or years, before a trigger finds them.
