Skip to content
GenAI Learn/Testing & Resilience in Production
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Disaster Recovery & Failover for AI Systems

4 min read

You'll learn to

  • -Translate an availability target into minutes of downtime per year
  • -Design a failover hierarchy and show with numbers why independent tiers beat a single provider
  • -Compute failover time from the health-check interval, and aim for degraded, never dark

An AI system powers real-time fraud detection for a bank. If it goes down, fraudulent transactions go through completely unchecked, with real financial consequences building up every second. This is the chapter's closing case brief, and it sets the bar: 99.99% availability, with graceful degradation even during a provider outage, a model failure or an infrastructure problem, not a best-effort recovery sometime later.

An Analogy: Backup Power

A hospital does not rely on the mains alone. If the grid fails, a generator starts. If the generator fails, batteries keep key equipment running. If those run out, staff use manual procedures. Each layer is slower or smaller than the one before, but the hospital is never completely in the dark. A failover hierarchy for an AI system works the same way.

A Failover Hierarchy, Not a Single Point of Failure

No single provider, however reliable, is a plan. The case brief's hierarchy has five tiers: the primary cloud provider, a secondary provider on genuinely different infrastructure, a locally self-hosted model as an always-available fallback, a semantic cache that returns a similar earlier answer as a last resort, and, below even that, simple rule-based logic with no AI at all.

Toggle which tiers are down below and watch the cumulative failover time recompute live, at a fixed 5-second health-check interval per tier.

Every Tier Down Adds One Health-Check Interval

Toggle which tiers are down and watch cumulative failover time recompute, live, at a 5-second health-check interval per tier.

Primary (cloud API)t = 0s
Secondary (different provider)t = 5s
Local model (self-hosted)t = 10s

Total failover time: 10s

At 50,000,000 transactions/day, that gap affects roughly 5,787 transactions, not the whole day.

For 99.99% uptime, the math calls for at least three genuinely independent failure modes stacked this way. If the primary, the secondary and the local model all fail at once, that overlap of independent failures is what the remaining 0.01% accounts for. Even then, the cache and the rule-based fallback keep the system minimally working instead of fully dark.

Health Checks Set the Actual Latency Budget

Every tier that has to be detected as down before the system moves on adds one full health-check interval to the total failover time. A health check is a periodic test that asks whether a tier is working. A 5-second interval with two unhealthy tiers ahead of the surviving one means roughly 10 seconds of degraded or unavailable service before full recovery. The interval itself, not only which fallback exists, is a real design decision with a real cost. The demo does the arithmetic for availability and for failover time.

Press Run: one provider allows hours of downtime a year, three independent tiers almost none

Degraded Is Not the Same as Down

A well-designed hierarchy means the system is never fully unavailable, only degraded: briefly slower, or briefly served by a less capable tier while it recovers. For the bank's fraud system, "temporarily using a smaller local model" is an acceptable few seconds. "No fraud checking is happening at all" is not acceptable for any length of time. Designing the hierarchy so the worst case is always the former is the actual engineering goal.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Build the A/B Tester and Disaster Recovery levels in the GenAI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!