Multi-Region & Active-Active Design

The availability chapter introduced active-active as a way to survive failure: several copies all serving traffic, so no single loss takes the system down. At global scale the same pattern gets a second job. It is not only about surviving a failure but about keeping delay low for users wherever they are, because distance itself costs time, and no amount of engineering inside one region can remove it.

Light in optical fiber travels at about 200,000 kilometers a second. A user in Mumbai talking to a server in Virginia, roughly 17,000 km of cable away, spends 17,000 / 200,000 = 85 ms just for the signal to arrive one way, and about 170 ms for a round trip, before the server has done any work. A single page often needs several round trips. The only cure is to put a copy of the system near the user, which is why large products run in several regions at once.

A global bank has a branch in Tokyo, one in Frankfurt and one in New York. Customers use the nearest one, which is quick and keeps working even if another branch is closed. Each branch keeps its own copy of the ledger and sends updates to the others as they happen. The trouble starts when the same account is changed in two branches at nearly the same moment, before either has heard from the other.

• -Each branch is a region, serving its nearby customers.
• -Sending ledger updates between branches is replication between regions.
• -The same account changed in two branches at once is a write conflict.
• -The rule for deciding which change wins is conflict resolution.

Running in several regions only helps if each user actually lands in a nearby one. Two techniques do that job. GeoDNS changes the answer to "what is the server's address?" depending on who asks. Anycast gives every region the same address and lets the internet's routers pick the nearest. Each is easy to picture with a familiar analogy, starting with the shopkeeper from the DNS chapter.

Recall the lookup from the DNS chapter. You ask a shopkeeper, the recursive resolver, for a company's phone number. He calls the company's own switchboard, the authoritative name server, writes the answer on a note by his till, and tells you. An ordinary switchboard gives every caller the same number. A GeoDNS switchboard has one extra habit: before it answers, it checks where the call is coming from and gives the number of the nearest branch.

• -The company has three branches, in Asia, Europe and the US. Each branch is a **region**, and its phone number is that region's IP address.
• -The switchboard holds all three numbers for the one name, plus a rule for which caller gets which. That is the **GeoDNS name server**.
• -The caller the switchboard sees is the shopkeeper, not you. It can only guess your location from his.
• -The note by the till is the cache, and the **TTL** says how long the note stays valid.

A worked example, using made-up addresses from a range reserved for documentation. The GeoDNS server for `shop.example.com` holds three addresses: `198.51.100.10` for Asia, `198.51.100.20` for Europe and `198.51.100.30` for the US. Priya in Tokyo opens the site. Her provider's resolver is in Tokyo too, so the switchboard sees a Tokyo caller and answers `198.51.100.10`, and her browser connects straight to the Asia region. Hans in Berlin asks about the same name, his resolver is in Berlin, and he gets `198.51.100.20`. One name, two different answers, each pointing at a nearby region.

GeoDNS has two weak spots, and both come back to the shopkeeper. First, it locates the resolver, not the person. If Hans uses a public resolver that answers from a data center on another continent, the switchboard sees a faraway caller and sends him to a faraway region. Some resolvers pass along a hint about the user's network to avoid this, and some hold it back on purpose for privacy.

Second, answers are cached. Suppose the Asia region fails at 10:00:00 and the switchboard stops handing out its address at once. A resolver that cached that answer at 9:59:50, with a TTL of 300 seconds, keeps giving out the dead address until 10:04:50. For almost five minutes some Tokyo users are sent to a region that is down. A shorter TTL narrows the window but cannot close it, because not every cache obeys the TTL it is given.

Now think of a national emergency number such as 911 in the United States. Everyone dials the same three digits, wherever they are, and nobody looks up a different number for their town. The phone network connects each call to the nearest emergency call center. If one center loses power, the network quietly sends calls to the next nearest one. Callers change nothing, and no cached note has to expire first.

• -The shared number is one IP address that every region uses at the same time.
• -Each call center is a **location**: a data center that can answer for that address.
• -The phone network is the internet's own routers. Each location announces to its neighboring routers that traffic for the address can be delivered through it, and the routers pass the news along. The protocol for this is called **BGP**.
• -Every router sends a packet along the shortest route it knows. That picks the nearest location by network distance, which is usually, but not always, the nearest on a map.

A worked example: the address `203.0.113.50` is announced from locations in Tokyo, Frankfurt and Chicago. Priya's packets for `203.0.113.50` take the shortest route and arrive in Tokyo. Hans's packets for the very same address arrive in Frankfurt. Nobody was handed a different address: the address is identical and the network did the choosing. Now the Tokyo location fails and stops announcing the address. The routers drop that route, and within seconds to a few tens of seconds Priya's packets follow the next shortest route to another location. Her browser still uses the same address, and there is no cached answer to wait out.

Anycast has its own catch. Routes can change in the middle of a long connection, and the later packets may then arrive at a different location that knows nothing about the connection. So anycast suits short exchanges best, such as a DNS question that fits in one packet or a quick request to a CDN, which is why DNS providers and CDNs use it so widely. Services that keep long connections open need extra care.

The diagrams show them in turn: the answers a GeoDNS server hands out, then one anycast address reaching a different location for each user, and then what happens when one location stops announcing the address.

• -**Who picks the region:** with GeoDNS, the name server, from where the question seems to come from. With anycast, the routers, from the shortest network route.
• -**What the user is given:** with GeoDNS, a different address depending on where they are. With anycast, the same address everywhere.
• -**Recovering from a failure:** GeoDNS has to wait for cached answers to expire, often minutes. Anycast reroutes as soon as the routers learn the route is gone, usually within seconds.
• -**What to watch for:** GeoDNS locates the resolver, not the user. Anycast chooses by network distance, not map distance, and a route change can cut a long connection.

The two are not rivals. A product can use anycast so users reach a nearby front door quickly, and GeoDNS to choose among its own regions. In an interview, name the one you would pick and the reason: GeoDNS is simple and lets you steer traffic region by region, and anycast wins when recovering in seconds matters more.

Keeping regions in step has a price. If a write had to be confirmed by another continent before it was accepted, every write would wait at least that 170 ms round trip, and would fail whenever the link between regions was cut. So regions replicate asynchronously: a write is accepted locally and sent on afterwards. That keeps writes fast and available, but it means that for a moment each region has a slightly different view, and it is where conflicts come from.

• -Avoid conflicts by design: give each user a home region, and send all of that user's writes there. Other regions hold read copies. Conflicts almost never arise, and if the home region fails, another takes over. Many systems start here.
• -Last-writer-wins: when two writes to one record collide, keep the one with the later timestamp. It is simple, but it silently throws one update away. Alice sets her display name to Ali on her phone in Tokyo at 10:00:00.100, and to Alicia from a laptop in Berlin at 10:00:00.105. Alicia wins and Ali vanishes, and because clocks in different regions disagree by milliseconds, the later write may even be the wrong winner.
• -CRDTs (conflict-free replicated data types): data structures designed so that concurrent updates always merge to the same result, with no coordination. A like counter is the classic example: each region counts only its own likes, say Europe 3 and the US 5, and merging just adds the parts, so every region ends at 8 regardless of the order in which the updates arrive.

The right choice depends on the data. A profile name can tolerate last-writer-wins, a like count suits a CRDT, and a bank balance should not be written in two regions at all.