Skip to content
GenAI Learn/Prompting & Model Behavior
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Streaming Responses & Latency

5 min read

You'll learn to

  • -Explain that streaming improves perceived speed, time to first token, without making the model faster
  • -Compare Server-Sent Events with WebSockets and pick the default
  • -Show with a simulation how backpressure stops a fast producer flooding a slow consumer

Users stare at a blank screen for eight to fifteen seconds before they see any response, while a competitor shows words appearing in real time. The bounce rate on a first query is 40%. The model is not slower than the competitor's. The response is simply withheld until generation is completely finished, instead of being shown as it is produced. Streaming fixes exactly this, and it does so without making the model generate a single token faster.

An Analogy: Dishes Served as They Are Ready

A restaurant can wait until the whole table's food is ready and then bring everything at once. Or it can bring each dish as it comes off the stove. The meal takes just as long to cook either way, but in the second restaurant you are eating within minutes and do not feel the wait. Streaming is bringing the dishes as they are ready.

Perceived Latency vs. Actual Latency

An LLM generates its output one token at a time, each new token depending on everything generated so far. A non-streamed response waits for every token to finish, then delivers the completed answer in one payload. A streamed response sends each token, or a small batch, to the client the moment it is generated. Total generation time is the same either way, since the model does the same work. But time to first token, how long before the user sees anything at all, drops from the full response time to a fraction of a second. The demo uses a 400-token answer to put numbers on it.

Press Run: the same 10.25 seconds in total, but the first words arrive after a quarter of a second

Now try the interactive version, which streams the same answer at different speeds and shows when the first words appear.

Same Generation Time, Very Different Experience

Both responses take the same 6 seconds to fully generate. Watch when the user actually sees something.

Streamedfirst token at 0.25s

Non-Streamedfirst token at —s

waiting for the full response...

t = 0.00s / 6.00s

The Transport: Server-Sent Events vs. WebSockets

  • -Server-Sent Events (SSE): a simple, one-directional stream of events over an ordinary HTTP connection. The client opens a connection and the server pushes tokens as they are produced. Most LLM provider APIs use this natively, and it needs no special infrastructure.
  • -WebSockets: a full two-way connection, useful when the client also needs to send data mid-stream, such as a cancel signal or a follow-up clarification while the first answer is still generating. More infrastructure to run properly, but strictly more capable when two-way communication during generation is needed.

For the common case, an assistant streaming an answer to a user who is just reading, SSE is simpler, is supported directly by most provider SDKs (software development kits, the libraries that wrap an API), and is usually the right default. Reach for WebSockets when the interaction genuinely needs to flow both ways during generation.

Backpressure and Buffering

A model can generate tokens faster than a slow client connection can render them, or faster than a downstream consumer such as a text-to-speech system can keep up. Backpressure is the general mechanism for that mismatch: a slow consumer signals the producer to pause, instead of the producer flooding a buffer that grows until something runs out of memory. It is the same idea as for message queues in the HLD course, applied here to a stream of tokens instead of a stream of jobs. The demo compares an unbounded buffer with a bounded one.

Press Run: without a limit memory grows; with backpressure it stays bounded

Streaming interacts with error handling in a way non-streamed responses do not. If a call fails halfway, the user has already seen a partial response. A production system must decide what a mid-stream failure looks like: a visible error appended to the partial text, or a clean retry that discards it. Do not leave that as an accidental implementation detail.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Practice controlling model behavior directly: the Temperature Dial, Prompt Architect, Cost Calculator, and Stream Weaver levels in the GenAI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!