Disaster Recovery & Failover for AI Systems

An AI system powers real-time fraud detection for a bank. If it goes down, fraudulent transactions go through completely unchecked, with real financial consequences accumulating for every second the system is unavailable. This is the chapter's closing case brief, and it sets the bar correctly: 99.99% availability, with graceful degradation even during a provider outage, a model failure, or an infrastructure issue, not just a best-effort recovery sometime later.

No single provider, however reliable, is a plan. The case brief's hierarchy has five tiers: the primary cloud provider, a secondary provider on a genuinely different infrastructure, a locally self-hosted model as an always-available fallback, a semantic cache returning a similar previous answer as a last resort, and, below even that, simple rule-based logic with no AI involved at all.

Toggle which tiers are down below and watch the actual cumulative failover time recompute, live, at a fixed 5-second health-check interval per tier.

Every tier that has to be detected as down before the system moves on adds one full health-check interval to the total failover time, exactly the arithmetic the demo above computes live. A 5-second interval with two unhealthy tiers ahead of the eventual survivor means roughly 10 seconds of degraded or unavailable service before the system fully recovers, which is precisely why the interval itself, not just which fallback exists, is a real design decision with a real cost attached.

A well-designed failover hierarchy means the system is never fully unavailable, only ever degraded, briefly slower or briefly served by a less capable tier, while it recovers. For the bank's fraud detection system specifically, "temporarily using a smaller local model" is an acceptable few seconds. "No fraud checking is happening at all" is not acceptable at any duration, and designing the hierarchy so the worst case is always the former, never the latter, is the actual engineering goal here.