From Caching Content to Running Code at the Edge
CDNs started as dumb caches: store a static file, serve it to nearby users, reduce origin load. Edge computing flipped this model. Instead of just caching, run actual application logic at the edge. Cloudflare Workers (2018) and AWS Lambda@Edge let you deploy JavaScript functions to 300+ locations worldwide. Each function runs within milliseconds of any user, globally.
The use cases expanded dramatically. Authentication checks that previously required a round-trip to your origin server can now run at the nearest edge node. A/B testing can assign users to variants at the edge before they ever reach your application. Rate limiting can be enforced globally without centralizing the state. Personalization based on country, device, or user segment can happen before the first byte of your application processes the request.
Edge computing shifts the question from 'where does my server live?' to 'where does my computation live?' Any logic that does not require your private data can potentially run closer to the user.
Multi-Region Active-Active (2020)
Traditional multi-region architecture meant active-passive: one region handles production traffic, another sits idle as a hot standby. Failover took minutes, sometimes required manual intervention, and frequently made things worse (failover bugs are the most spectacular kind). Active-active changes the model: multiple regions simultaneously serve production traffic.
With active-active, users are routed to their nearest region via GeoDNS or anycast. Each region has a complete copy of the data, replicating asynchronously to other regions. The benefit: if a hurricane takes out us-east-1, traffic automatically fails over to other regions with zero manual intervention and zero customer-visible downtime.
The engineering challenge is conflict resolution. When two users in different regions modify the same record simultaneously, who wins? Solutions depend on the data type:
For counters and accumulators: CRDTs provide conflict-free convergence
For append-only data (logs, events): no conflicts possible by design
For last-writer-wins with logical timestamps: simplest, but loses one update on conflict
For application-level merge: complex but highest fidelity. The application defines what 'merge two conflicting versions' means for its domain.
For financial data: synchronous replication to at least one other region before acknowledging writes. Slightly slower, but correct.
Geo-Sharding and Data Residency (2023)
GDPR in Europe, LGPD in Brazil, PIPL in China. Data residency laws created a constraint that breaks traditional multi-region replication: certain data legally cannot leave certain geographic boundaries. A European user's personal data must live on European servers and must not replicate to US data centers. Traditional active-active assumes data flows freely between regions. Data residency laws say it cannot.
Geo-sharding addresses this by partitioning data based on user geography. European user data lives on eu-west servers and stays there. US user data lives on us-east servers. Each regional shard is independent from a data sovereignty perspective. The complexity emerges at the edges: what happens when a European user travels to the US? What happens when global analytics need to aggregate across regional shards? What happens when a user changes their home country?
Read-local, write-local: users always interact with their home region's data. Travel creates a temporary degraded experience (slightly stale reads from home region).
Data forwarding: certain queries are forwarded to the user's home region regardless of where the request originated.
Anonymized cross-region analytics: aggregate statistics without including individual user records. GDPR-compliant global dashboards.
Region-pinned replicas: Spanner and CockroachDB support specifying which regions can hold replicas for specific data, enforcing residency at the database level.
Further Reading
Cloudflare Workers documentation for edge computing patterns
AWS re:Invent talks on multi-region active-active architecture
GDPR technical guidance for data residency implementation patterns
Cockroach Labs blog on multi-region database design for data residency
GeoDNS and Anycast: Routing Traffic to the Right Region
Two mechanisms route users to geographically appropriate servers. GeoDNS (Geographic DNS) returns different IP addresses based on the requester's location. A DNS query from a Tokyo IP gets the Tokyo load balancer's address. A query from Berlin gets the EU load balancer. GeoDNS is simple to operate but has a significant lag: DNS TTLs mean a user traveling from Tokyo to Berlin continues hitting the Tokyo data center until their DNS cache expires (potentially hours). Anycast is different: the same IP address is announced from multiple data centers simultaneously using BGP. The internet's routing protocol automatically directs packets to the 'nearest' (fewest BGP hops) location. Traffic fails over automatically when a location goes down, and there is no DNS TTL lag. Cloudflare's entire global network uses anycast.
GeoDNS: DNS resolver returns region-specific IP. Simple. Affected by DNS caching lag on region change.
Anycast: same IP, multiple origin locations. BGP routes to nearest. Instant failover, no DNS TTL issues.
Latency-based routing (AWS Route 53): measures actual latency from the client to each region and routes to the lowest-latency option.
Geofencing for compliance: ensure EU users are always routed to EU regions regardless of latency. Required for GDPR data residency.
Health check routing: remove a region from rotation when health checks fail. Route 53 and Cloudflare both support automated failover.
Testing Multi-Region Systems: Failure Drills and Chaos
A multi-region architecture that has never been tested under failure is a theoretical resilience improvement, not an operational one. Failover procedures that look correct in runbooks often fail in practice due to stale DNS caches, database replication lag exceeding expected bounds, or application code that was written assuming a single-region topology. The only way to know your failover works is to practice it. Netflix's Chaos Kong simulates full regional outages regularly. AWS runs regular regional evacuation drills internally. The cadence matters: quarterly is the minimum; monthly is better. Each drill surfaces assumptions that turn out to be wrong.
Regional failover drill: block all traffic to a region and verify that the remaining regions absorb load within your SLO.
Replication lag measurement: under peak write load, what is the P99 replication lag? Does it stay within the staleness your SLO allows?
DNS TTL validation: verify that your TTLs allow traffic to shift quickly enough during failover. 60-second TTLs vs 5-minute TTLs matter during incidents.
Split-brain testing: simulate a network partition between regions and verify that conflict resolution logic produces correct outcomes.
Runbook validation: can a new on-call engineer successfully execute a regional failover using only the documented runbook? If not, improve the runbook.
What Interviewers Test About Global System Design
Global system design questions (design a system for a global user base, design a system compliant with GDPR) test whether you can reason about the interaction between network physics, data consistency requirements, and regulatory constraints. The best answers are specific about trade-offs: what consistency you sacrifice for latency, how you handle the conflict between data residency and operational simplicity.
Know: the speed-of-light RTT between major regions (US-EU ~80ms, US-Asia ~150ms, EU-Asia ~120ms). These are hard floors, not configuration parameters.
Know: the difference between multi-region active-passive (one region hot standby) and active-active (both regions serving traffic simultaneously)
Know: what data must be geo-restricted under GDPR and how geo-sharding enforces it at the database level
Know: how read-your-writes consistency is maintained for users who travel between regions
Know: the operational complexity cost of active-active (conflict resolution, cross-region latency for synchronous replication) vs the resilience benefit
GeoDNS and Anycast: Routing Traffic to the Right Region
Two mechanisms route users to geographically appropriate servers. GeoDNS (Geographic DNS) returns different IP addresses based on the requester's location. A DNS query from a Tokyo IP gets the Tokyo load balancer's address. A query from Berlin gets the EU load balancer. GeoDNS is simple to operate but has a significant lag: DNS TTLs mean a user traveling from Tokyo to Berlin continues hitting the Tokyo data center until their DNS cache expires (potentially hours). Anycast is different: the same IP address is announced from multiple data centers simultaneously using BGP. The internet's routing protocol automatically directs packets to the 'nearest' (fewest BGP hops) location. Traffic fails over automatically when a location goes down, and there is no DNS TTL lag. Cloudflare's entire global network uses anycast.
GeoDNS: DNS resolver returns region-specific IP. Simple. Affected by DNS caching lag on region change.
Anycast: same IP, multiple origin locations. BGP routes to nearest. Instant failover, no DNS TTL issues.
Latency-based routing (AWS Route 53): measures actual latency from the client to each region and routes to the lowest-latency option.
Geofencing for compliance: ensure EU users are always routed to EU regions regardless of latency. Required for GDPR data residency.
Health check routing: remove a region from rotation when health checks fail. Route 53 and Cloudflare both support automated failover.
Testing Multi-Region Systems: Failure Drills and Chaos
A multi-region architecture that has never been tested under failure is a theoretical resilience improvement, not an operational one. Failover procedures that look correct in runbooks often fail in practice due to stale DNS caches, database replication lag exceeding expected bounds, or application code that was written assuming a single-region topology. The only way to know your failover works is to practice it. Netflix's Chaos Kong simulates full regional outages regularly. AWS runs regular regional evacuation drills internally. The cadence matters: quarterly is the minimum; monthly is better. Each drill surfaces assumptions that turn out to be wrong.
Regional failover drill: block all traffic to a region and verify that the remaining regions absorb load within your SLO.
Replication lag measurement: under peak write load, what is the P99 replication lag? Does it stay within the staleness your SLO allows?
DNS TTL validation: verify that your TTLs allow traffic to shift quickly enough during failover. 60-second TTLs vs 5-minute TTLs matter during incidents.
Split-brain testing: simulate a network partition between regions and verify that conflict resolution logic produces correct outcomes.
Runbook validation: can a new on-call engineer successfully execute a regional failover using only the documented runbook? If not, improve the runbook.
What Interviewers Test About Global System Design
Global system design questions (design a system for a global user base, design a system compliant with GDPR) test whether you can reason about the interaction between network physics, data consistency requirements, and regulatory constraints. The best answers are specific about trade-offs: what consistency you sacrifice for latency, how you handle the conflict between data residency and operational simplicity.
Know: the speed-of-light RTT between major regions (US-EU ~80ms, US-Asia ~150ms, EU-Asia ~120ms). These are hard floors, not configuration parameters.
Know: the difference between multi-region active-passive (one region hot standby) and active-active (both regions serving traffic simultaneously)
Know: what data must be geo-restricted under GDPR and how geo-sharding enforces it at the database level
Know: how read-your-writes consistency is maintained for users who travel between regions
Know: the operational complexity cost of active-active (conflict resolution, cross-region latency for synchronous replication) vs the resilience benefit
Related reading
GeoDNS and Anycast: Routing Traffic to the Right Region
Two mechanisms route users to geographically appropriate servers. GeoDNS (Geographic DNS) returns different IP addresses based on the requester's location. A DNS query from a Tokyo IP gets the Tokyo load balancer's address. A query from Berlin gets the EU load balancer. GeoDNS is simple to operate but has a significant lag: DNS TTLs mean a user traveling from Tokyo to Berlin continues hitting the Tokyo data center until their DNS cache expires (potentially hours). Anycast is different: the same IP address is announced from multiple data centers simultaneously using BGP. The internet's routing protocol automatically directs packets to the 'nearest' (fewest BGP hops) location. Traffic fails over automatically when a location goes down, and there is no DNS TTL lag. Cloudflare's entire global network uses anycast.
- GeoDNS: DNS resolver returns region-specific IP. Simple. Affected by DNS caching lag on region change.
- Anycast: same IP, multiple origin locations. BGP routes to nearest. Instant failover, no DNS TTL issues.
- Latency-based routing (AWS Route 53): measures actual latency from the client to each region and routes to the lowest-latency option.
- Geofencing for compliance: ensure EU users are always routed to EU regions regardless of latency. Required for GDPR data residency.
- Health check routing: remove a region from rotation when health checks fail. Route 53 and Cloudflare both support automated failover.
Testing Multi-Region Systems: Failure Drills and Chaos
A multi-region architecture that has never been tested under failure is a theoretical resilience improvement, not an operational one. Failover procedures that look correct in runbooks often fail in practice due to stale DNS caches, database replication lag exceeding expected bounds, or application code that was written assuming a single-region topology. The only way to know your failover works is to practice it. Netflix's Chaos Kong simulates full regional outages regularly. AWS runs regular regional evacuation drills internally. The cadence matters: quarterly is the minimum; monthly is better. Each drill surfaces assumptions that turn out to be wrong.
- Regional failover drill: block all traffic to a region and verify that the remaining regions absorb load within your SLO.
- Replication lag measurement: under peak write load, what is the P99 replication lag? Does it stay within the staleness your SLO allows?
- DNS TTL validation: verify that your TTLs allow traffic to shift quickly enough during failover. 60-second TTLs vs 5-minute TTLs matter during incidents.
- Split-brain testing: simulate a network partition between regions and verify that conflict resolution logic produces correct outcomes.
- Runbook validation: can a new on-call engineer successfully execute a regional failover using only the documented runbook? If not, improve the runbook.
What Interviewers Test About Global System Design
Global system design questions (design a system for a global user base, design a system compliant with GDPR) test whether you can reason about the interaction between network physics, data consistency requirements, and regulatory constraints. The best answers are specific about trade-offs: what consistency you sacrifice for latency, how you handle the conflict between data residency and operational simplicity.
- Know: the speed-of-light RTT between major regions (US-EU ~80ms, US-Asia ~150ms, EU-Asia ~120ms). These are hard floors, not configuration parameters.
- Know: the difference between multi-region active-passive (one region hot standby) and active-active (both regions serving traffic simultaneously)
- Know: what data must be geo-restricted under GDPR and how geo-sharding enforces it at the database level
- Know: how read-your-writes consistency is maintained for users who travel between regions
- Know: the operational complexity cost of active-active (conflict resolution, cross-region latency for synchronous replication) vs the resilience benefit
