Multi-Master Replication: Writing to Multiple Servers Simultaneously

How Google Docs Handles 100 Million Simultaneous Editors

When two people edit the same Google Doc at the same time, both see each other's changes in real-time. Behind the scenes, both users are writing to different servers. Google uses a form of multi-master replication with Operational Transformation (OT) to merge conflicting edits. When User A types 'Hello' at position 5 and User B deletes a word at position 3, both operations get transformed so the final document ends up consistent on both servers. This is the hardest problem in multi-master replication: making concurrent writes converge to the same result everywhere, without waiting on coordination delays.

Single-Master vs Multi-Master

Single-Master (Primary-Replica):

  Tokyo User --> 150ms --> [Primary in Virginia] --> writes
                          [Replica in Tokyo]     --> reads (fast)

  All writes go to Virginia. Tokyo users have 150ms write latency.
  If Virginia dies: promotion takes 30-300 seconds (downtime).

Multi-Master:

  Tokyo User --> [Master Tokyo]  --> writes locally (5ms)
  NYC User   --> [Master Virginia]--> writes locally (5ms)

  Both masters replicate to each other asynchronously.
  If Tokyo master dies: Virginia continues without interruption.

  But: what if both users update the same row at the same time?
  Tokyo: UPDATE users SET name='Tanaka' WHERE id=42
  Virginia: UPDATE users SET name='Smith' WHERE id=42

  CONFLICT! Which name wins? This is the fundamental problem.

This tradeoff shows up in almost every distributed database system once you scale past a single region, and it's one of the core patterns worth understanding if you're working through the building blocks of system design.

Conflict Resolution Strategies

Conflict Resolution Approaches:

  1. Last-Writer-Wins (LWW):
     Use timestamp to pick the later write.
     Simple, but SILENTLY LOSES DATA.
     Tokyo writes at 10:00:00.001, Virginia at 10:00:00.002
     Virginia wins. Tokyo's write disappears without error.
     Danger: clock skew between data centers!

  2. Application-Level Merge:
     Shopping Cart Conflict:
     Master A cart: {item1, item2, item3}
     Master B cart: {item1, item2, item4}
     Merge: UNION -> {item1, item2, item3, item4}

     But numeric conflicts cannot union:
     Master A: balance = $100 (deposited $50)
     Master B: balance = $80  (withdrew $20)
     Merge: ??? (both started at $50)
     Need: event log, not snapshot. Replay: +50, -20 = $80.

  3. CRDTs (Conflict-free Replicated Data Types):
     Mathematically guaranteed convergence.
     G-Counter: each replica has its own counter.
     Total = sum of all replica counters.
     NEVER conflicts. Used by Redis, Riak, Figma.

  4. Conflict Avoidance (Partition by Owner):
     User 42 'lives' on Tokyo master.
     All writes for user 42 go to Tokyo.
     Virginia has a read-only replica.
     No conflicts possible for user-scoped data.

Notice that the balance example only works if you keep an event log instead of overwriting a snapshot. That's the same principle behind event sourcing, where you store the sequence of things that happened rather than just the current state. If that idea is new to you, it's worth reading how Kafka and event sourcing apply it outside the replication context too.

Multi-Master Technologies

CockroachDB takes a different path from the rest of this list. It uses Raft consensus for every write, so there are no conflicts by design, writes are simply serialized. The cost is higher write latency from the consensus round-trip, but you get strong consistency even across a multi-region deployment. It's a good example of trading raw write speed for correctness guarantees, similar to how you'd weigh cache placement tradeoffs against staleness risk.

When to Use Multi-Master

• Global applications needing under 50ms write latency across multiple continents. Single-master means 100-200ms writes from distant regions.
• Extreme availability requirements. Single-master failover takes 30-300 seconds, while multi-master gives you zero downtime on node failure.
• Collaborative editing where multiple users modify shared data simultaneously, like Google Docs or Figma.
• Do NOT use it for most applications. Single-master with read replicas is simpler and covers the vast majority of use cases. The conflict resolution complexity of multi-master only pays off at global scale or under strict high-availability requirements.

Interview Tip

Multi-master comes up when designing global systems. Start by explaining the latency problem that motivates it: single-master means 150ms+ writes for distant users. Then immediately discuss conflict resolution, since that's the part that shows you actually understand the hard problem. Present three strategies: LWW (simple but lossy), application-level merge (domain-specific), and CRDTs (mathematically safe). Mention conflict avoidance as the best approach when possible: partition data so each record has a home master. The key phrase to land: 'multi-master trades consistency complexity for write latency and availability.' Always mention CockroachDB as the modern approach, since it gives multi-region writes with serializable consistency.

Key Takeaway

Multi-master replication enables low-latency writes in multiple regions and zero-downtime failover. The cost is conflict resolution complexity, you have to choose between LWW (data loss risk), application-level merging (domain-specific complexity), or CRDTs (limited data types). For most applications, single-master with read replicas is simpler and sufficient. Reach for multi-master only when global write latency or extreme availability actually demand it.

If you want to keep working through related distributed systems tradeoffs, the writeup on API gateways and multi-protocol architecture covers a similar theme: how much complexity you take on at the edge versus in the data layer depends entirely on what your traffic actually looks like.