API Design Learn/Real-Time & Streaming APIs

Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Choosing a Real-Time Transport

6 min read

You'll learn to

-Compare polling, long polling, Server-Sent Events and WebSockets by direction, overhead and reconnection
-Design resume so a client that drops and reconnects misses no events
-Use heartbeats to detect dead connections, and backoff with jitter to avoid a reconnect stampede
-Pick the simplest transport that fits a given product

Plain HTTP is a question and an answer: the client asks, the server replies, and the conversation ends. A live scoreboard, a chat window or a presence dot needs the opposite, where the server speaks first. There are four common ways to get there, and they differ in direction, cost and how well they survive a bad network.

An Analogy: Four Ways to Get the News

Polling is walking to the mailbox every minute to check. Long polling is asking the clerk for news and having them stand there until something arrives, then asking again. Server-Sent Events is tuning in to a one-way radio broadcast. A WebSocket is a phone call, where either side can talk at any time. Each is right for a different job.

The Four Options

-Polling: the client repeats GET /scores every few seconds. Simple and cache-friendly, but most answers are 'nothing new' and updates arrive late by up to one interval.
-Long polling: the server holds the request open until there is something to send or a timeout of about 30 seconds passes. Updates are prompt and it works everywhere, but every event costs a new request.
-Server-Sent Events (SSE): one long-lived HTTP response that streams events one way, server to client. Browsers reconnect on their own and send the last event id they saw.
-WebSocket: an HTTP request upgraded into a two-way connection (RFC 6455). Lowest overhead per message, and the right tool when the client talks as much as the server.

Direction is the first question. If only the server pushes, SSE is usually enough and plays well with proxies, caches and ordinary HTTP authentication. If both sides send frequent messages, as in chat or collaborative editing, use a WebSocket. Long polling is the fallback for networks that block everything else, and polling is fine when updates are rare.

The three pictures below show the same scoreboard growing up. First the client keeps asking, then the server pushes, then a fleet of push servers shares one stream of events, so a viewer can be connected to any of them.

1Polling: the client keeps asking

Browser

API server

Most answers are 'nothing new', and an update can wait up to one interval.

BrowserAPI server- GET /scores every 5 seconds

2SSE or WebSocket: the server pushes

API server

Browser

One long-lived connection. Each event carries an increasing id.

API serverBrowser- event 12, 13, 14 as they happen

3Many servers: publish once, deliver anywhere

Score service

Event bus

Push server 1

Push server 2

Viewer on server 1

Viewer on server 2

Any push server can serve any viewer, because the events come from a shared bus.

Score serviceEvent bus- publish oncePush server 1Viewer on server 1- streamPush server 2Viewer on server 2- stream

Resuming After a Drop

Mobile networks drop connections constantly, so a real-time API must assume the client will miss events. The fix is to give every event an increasing id. The client remembers the last id it processed and sends it when it reconnects: SSE does this automatically with the Last-Event-ID header, and a WebSocket client sends it in its first message. The server keeps a short replay buffer and sends only what was missed.

The buffer cannot be infinite. If the client has been away longer than the buffer reaches, the server cannot fill the gap, and it should say so, for example with a resync message, so the client refetches the full state from a normal endpoint and then continues from the stream. The block below shows both cases, followed by a reconnect schedule that uses backoff with jitter. Press Run and read the delays.

Press Run: a client replays what it missed, a stale client is told to resync, and reconnect waits grow with jitter

import random

log = []                              # the server's replay buffer: (event id, payload)
def publish(payload):
    log.append((len(log) + 1, payload))

def replay_since(last_id, keep=100):
    oldest = log[0][0] if log else 1
    if last_id < oldest - 1 or last_id < len(log) - keep:
        return "resync"               # the gap is older than the buffer: refetch the full state
    return [e for e in log if e[0] > last_id]

for score in ["1-0", "1-1", "2-1", "2-2", "3-2", "3-3"]:
    publish("score " + score)

last_seen = 3                         # the client saw events 1-3, then lost the connection
print("reconnect with Last-Event-ID 3 ->", replay_since(last_seen))
print("a client that was away too long ->", replay_since(0, keep=2))

rng = random.Random(7)                # backoff with full jitter: wait a random time up to a cap
for attempt in range(1, 7):
    cap = min(30, 2 ** attempt)
    print("attempt", attempt, "cap", cap, "s, waits", round(rng.random() * cap, 1), "s")

import random

log = []                              # the server's replay buffer: (event id, payload)
def publish(payload):
    log.append((len(log) + 1, payload))

def replay_since(last_id, keep=100):
    oldest = log[0][0] if log else 1
    if last_id < oldest - 1 or last_id < len(log) - keep:
        return "resync"               # the gap is older than the buffer: refetch the full state
    return [e for e in log if e[0] > last_id]

for score in ["1-0", "1-1", "2-1", "2-2", "3-2", "3-3"]:
    publish("score " + score)

last_seen = 3                         # the client saw events 1-3, then lost the connection
print("reconnect with Last-Event-ID 3 ->", replay_since(last_seen))
print("a client that was away too long ->", replay_since(0, keep=2))

rng = random.Random(7)                # backoff with full jitter: wait a random time up to a cap
for attempt in range(1, 7):
    cap = min(30, 2 ** attempt)
    print("attempt", attempt, "cap", cap, "s, waits", round(rng.random() * cap, 1), "s")

The first line shows the replay: events 4, 5 and 6 come back and nothing is repeated. The second shows the resync answer. In the schedule, each wait is a random value between zero and the cap, and the cap doubles up to 30 seconds. Because every client picks a different random wait, a server that restarts and drops 100,000 connections does not see 100,000 reconnects in the same instant.

Try it yourself in the demo below. Change how long the client was away and how many events the server keeps, and watch the answer switch between a replay and a resync at exactly the point where the gap outgrows the buffer.

Reconnect: Replay or Resync?

A phone loses signal and comes back. Every event had an increasing id, and the server keeps a short replay buffer. Move the sliders to see when a replay is enough and when the client has to start over from the full state.

Events per second on this stream2

Seconds the client was away20 s

Events the server keeps in its replay buffer100

Events missed

2 per second for 20 seconds

Server answers

Replay 40

from Last-Event-ID

Longest outage covered

50 s

buffer size divided by event rate

Buffer used by the replay40 of 100

Reconnect waits (random, up to): 2 s, 4 s, 8 s, 16 s, 30 s, 30 s. Each client picks its own random value, so a restart does not bring every client back at once.

The client sends the id of the last event it saw. The server finds 40 newer events in its buffer and replays exactly those, in order, so the screen is right again with no reload.

Heartbeats and Dead Connections

A connection can die without either side noticing: a phone enters a tunnel, a NAT table forgets the mapping, a proxy closes an idle stream. TCP may not report it for minutes. So the server sends a small heartbeat every 15 to 30 seconds (an SSE comment line, or a WebSocket ping), and the client treats two missed heartbeats as a dead connection and reconnects. The same heartbeat also stops idle-timeout rules in proxies from closing a quiet stream.

Running It at Scale

-Each open connection holds a socket and memory, so one server may carry tens of thousands, and you size for connections as well as requests.
-Keep no per-user state that only one server knows. Publish events through a shared bus so any server can push to any connected client.
-Authenticate when the connection opens, and plan for the token to expire while the connection is still up: re-authenticate or close with a clear code.
-Protect the server from slow clients with a bounded send buffer. When it fills, drop the oldest events or disconnect the client, and let resume repair the gap.
-Over HTTP/1.1, browsers allow only about six connections per domain, which a few open event streams can use up. Serve streams over HTTP/2, where many of them share one connection.

Pick the simplest transport that does the job: SSE for one-way pushes such as scores, notifications and progress, a WebSocket for two-way traffic such as chat and shared editing, and long polling only where the network forces it. Whichever you choose, event ids, resume, heartbeats and jittered reconnect are what make it dependable.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Scoreboard Live, Long-Poll Lifeline and Socket Sync in the API Design Lab.

Dead Letter Queues & Retry Policies Ordering Keys, Partitions and Replay

Discussion0

Join the Discussion

No comments yet. Be the first to start the thread!