WebSocket Architectures at Scale: Handling Millions of Persistent Connections

Discord's 10 Million Concurrent Problem

Discord handles over 10 million concurrent WebSocket connections. Each user has an open connection that receives messages, typing indicators, presence updates, and voice state changes in real-time. But here is the hard part: User A on Server 37 sends a message to a channel where User B is connected to Server 194. How does that message cross 157 servers to reach the right person in under 50 milliseconds?

Architecture: The Three-Layer Stack

WebSocket Architecture at Scale:

  [Millions of Clients]
         |
  [Load Balancer (L4, sticky sessions)]
         |
  +------+------+------+------+
  |      |      |      |      |
  [GW-1] [GW-2] [GW-3] [GW-4] ...  <-- Gateway Layer
  50K    50K    50K    50K         (holds WS connections)
  conns  conns  conns  conns
  |      |      |      |
  +------+------+------+------+
         |
  [Redis Pub/Sub or Kafka]          <-- Message Bus
         |
  +------+------+------+
  |      |      |      |
  [App1] [App2] [App3]              <-- Application Layer
  (stateless business logic)        (auth, store, filter)

  Gateway: stateful (tracks who is connected where)
  App:     stateless (any instance handles any request)
  Bus:     routes messages between gateways

The Cross-Server Message Routing Problem

User A (Gateway 1) sends message to #general channel:

  1. Gateway 1 receives WS message from User A
  2. Gateway 1 sends to App Service (HTTP/gRPC)
  3. App Service: validates, stores in DB, publishes to bus
  4. App Service publishes to Redis: PUBLISH channel:#general {msg}
  5. ALL gateways subscribe to channel:#general:
     - Gateway 1: has Users A, C, F in #general -> push to them
     - Gateway 2: has Users B, D in #general -> push to them
     - Gateway 3: nobody in #general -> ignores
     - Gateway 4: has User E in #general -> push to E

  Total latency: ~5-15ms (mostly Redis pub/sub overhead)

Connection Limits and Tuning

Per-Server Connection Limits:

Resource          Default    Production Target   Command
----------------  ---------  ------------------  ----------------
File descriptors  1,024      1,000,000+          ulimit -n 1048576
TCP memory/conn   ~10-50 KB  (~50 GB for 1M)     sysctl tuning
Ephemeral ports   ~28,000    65,535 max          net.ipv4.ip_local
Threads           varies     async I/O needed    Use epoll/kqueue

Practical guidance:
  50,000 connections per server  = comfortable
  100,000 connections per server = optimized
  1,000,000 connections per box  = possible with C/Rust + io_uring

Heartbeat: ping/pong every 30s
  - Detects dead connections (mobile went to sleep)
  - Frees resources from zombie connections
  - Without heartbeats: memory leak from undead connections

Deployments: The Graceful Drain Problem

Deploying with sticky WS connections:

  WRONG (instant restart):
  1. Kill Gateway-1 -> 50,000 users disconnected
  2. All 50K reconnect simultaneously (thundering herd)
  3. Other gateways overwhelmed -> cascading failure

  RIGHT (graceful drain):
  1. Remove Gateway-1 from load balancer
     (no NEW connections)
  2. Send 'reconnect' signal to connected clients
     (stagger: 1000/sec over 50 seconds)
  3. Clients reconnect to other gateways automatically
  4. Wait until Gateway-1 has 0 connections
  5. Shut down Gateway-1 safely
  6. Deploy new version and re-add to load balancer

Interview Tip

When designing a chat or real-time system, say: 'I would separate stateful WebSocket gateways from stateless application servers. Gateways hold connections and track which users are connected. A pub/sub bus (Redis) routes messages between gateways. For 10M users, I need ~200 gateway servers at 50K connections each. I would use sticky sessions at the L4 load balancer, heartbeats every 30 seconds to detect dead connections, and graceful draining for zero-downtime deployments.'

<

/blockquote>

Key Takeaway

Scale WebSocket systems with a stateful connection layer backed by a pub/sub bus for cross-server messaging. Use sticky sessions at the load balancer, async runtimes for connection density, and heartbeats to clean up dead connections.