Streaming Responses & Latency

Users stare at a blank screen for eight to fifteen seconds before seeing any response at all, while competitors show tokens appearing in real time, and the bounce rate on a first query sits at 40%. The model is not actually slower than the competitor's. The response is simply being withheld from the user until generation finishes completely, instead of being shown as it is produced. Streaming fixes exactly this, and it fixes it without making the model generate a single token faster.

An LLM generates output one token at a time, autoregressively, each new token conditioned on everything generated so far. A non-streamed response waits for every single token to finish, then delivers the entire completed response in one payload. A streamed response instead sends each token (or small batch of tokens) to the client the moment it is generated. Total generation time is identical in both cases, the model does the same amount of work either way, but time to first token, the time before the user sees anything at all, drops from "the full response time" to "a fraction of a second." That single change in perceived latency is what closes the gap in the bounce-rate story above.

• -Server-Sent Events (SSE): a simple, one-directional stream of events over a regular HTTP connection, the client opens a connection and the server pushes tokens as they are produced. This is what most LLM provider APIs use natively for streaming, and it needs no special infrastructure beyond standard HTTP.
• -WebSockets: a full-duplex connection, useful when the client also needs to send data back mid-stream (an interrupt/cancel signal, a follow-up clarification while the first answer is still generating). More infrastructure to run correctly, but strictly more capable when bidirectional communication during generation is actually needed.

For the common case, an assistant streaming a response to a user who is just reading, SSE is simpler, is supported directly by most LLM provider SDKs, and is usually the right default. Reach for WebSockets specifically when the interaction genuinely needs to flow both directions during generation.

A model can generate tokens faster than a slow client connection can render them, or faster than a downstream consumer (a text-to-speech pipeline, a UI animation) can keep up. Backpressure is the general mechanism for handling that mismatch: a slow consumer signals the producer to pause, rather than the producer blindly flooding a buffer that keeps growing until something runs out of memory, the exact same concept covered for message queues back in the HLD course, just applied here to a stream of tokens instead of a stream of jobs.