Health Checks, Graceful Draining and Multi-Region Routing

In a healthy fleet, servers are replaced all day: deploys, autoscaling, failed machines, maintenance. Clients should never notice. That takes two things from the API: honest signals about whether each server can take traffic, and a polite way to leave. A third problem is bigger: when a whole region is lost, traffic has to go somewhere else.

When a section of a restaurant closes, the host stops seating new tables there, the staff let current diners finish, and only then is the section locked. The host knows which sections are open, so new guests are sent elsewhere. A load balancer is the host, and health checks are how it learns which sections are open.

• -**Liveness:** 'is this process wedged?' If it fails, the platform restarts the process. Keep it cheap and independent of other systems. If it also checks the database, a database outage makes every server fail liveness and restart together, which turns an incident into a larger one.
• -**Readiness:** 'can this server take traffic right now?' It checks what the server needs to do its job: warmed up, dependencies reachable, not draining. If it fails, the load balancer stops sending requests, but the process is left running and recovers on its own.
• -**Startup:** 'has it finished booting?' It gives slow starters time before liveness begins to count, so a heavy start is not mistaken for a hang.

The pictures show a healthy fleet, then what readiness does to one server that is leaving, then what a global router does when a whole region fails its health check.

A server that is simply killed drops every request it was serving. Draining is the polite exit: it begins when the platform sends a termination signal and has a fixed order.

• -Fail readiness, so the load balancer stops sending new requests.
• -Wait a few seconds, because the balancer takes a moment to notice.
• -Let in-flight requests finish, up to a grace period.
• -Close idle keep-alive connections (`Connection: close`, or an HTTP/2 GOAWAY frame) and tell WebSocket and SSE clients to reconnect, with jitter.
• -Exit. The grace period must be longer than the slowest normal request, or the platform kills the server in the middle of work.

The simulation below follows a server with three requests in flight and a grace period of 8 seconds. Press Run, then change `GRACE` to 5 and run it again to see what a grace period that is too short does.

With 8 seconds, the slowest request (req-2, which needs 5 more seconds) finishes at t=7 and the exit is clean. With 5 seconds, req-2 is still running when time is up and is cut off, which is exactly the failure that a long enough grace period and a client retry policy are meant to avoid. A request that was cut off should be safe to retry, which is another reason for idempotency keys.

Use the demo below to find a safe grace period. Slow the busiest request down or make the load balancer slower to notice, and see which requests the platform would cut off, and when the process could have exited cleanly.

To survive the loss of a region and to serve users nearby, run the API in several regions and put a global layer in front (geographic DNS, anycast, or a global load balancer). It sends each user to the closest healthy region, using latency or location, and its own health checks remove a region that fails. Press Run to see routing and failover.

Routing is the easy half. The data is the hard half. Reads can be served from a nearby replica, but writes need a rule: either each entity has a home region that takes its writes (simple and consistent, but far users pay latency and a lost home region needs a promotion plan), or several regions accept writes and conflicts are resolved by rules such as CRDTs. DNS answers are cached, so failover takes as long as the record's time to live unless an anycast or global load balancer reacts sooner.

• -A retry after a regional failure may reach a different region, and the first attempt may have succeeded. Idempotency keys must be honored across regions, or the retry double-charges.
• -Keep session and rate limit state in a place that survives a region loss, or accept that users start over after failover.
• -Rehearse failover. A runbook that has never been run is a hope, not a plan.