Response streaming· LLM delivery
Response Streaming — Letting Customers Read While the Model Writes
Quick answer: Without streaming, a chatbot's server waits until the model has produced the whole answer and then returns it in one piece, so a long reply means a long blank wait. With streaming, the server forwards each fragment (usually a few tokens) the moment it exists, and the chat widget draws it straight away. Total generation time does not shrink; what shrinks is the wait before anything appears, which is the number measured as time to first token.
How a streamed answer travels
The usual path has three hops. The browser asks your server for an answer. Your server asks the model provider with streaming switched on. The provider sends back a series of small events, and your server passes the useful ones on to the browser, which appends each piece to the message on screen.
On the provider side, OpenAI's Responses API turns streaming on with a stream: true setting, and its guide says the stream is delivered over server-sent events. Each event has a type; for plain text the guide points you to response.created, response.output_text.delta (a fragment of text), response.completed and error. Other providers use their own event names, so treat these as one documented example, not a standard.
Server-sent events in one paragraph
Server-sent events (SSE) are a web feature where the server keeps one HTTP response open and writes messages into it. Per MDN, the connection is one-way, from server to browser; the response uses the content type text/event-stream; each message is a few field: value lines ended by a blank line; and the browser's EventSource object reconnects by default if the connection drops. MDN also notes a limit that matters for chat: without HTTP/2, browsers allow only six SSE connections per browser and domain, shared across tabs. Plain EventSource also only sends GET requests, so chat widgets often read the stream from a fetch call instead.
Streaming, typing indicators and total time
These three get confused, because all of them are about waiting.
| Concept | What it does | Does the wait end sooner? |
|---|---|---|
| Response streaming | Shows real text as it is generated | Yes, the first words appear sooner; the last word arrives at the same time |
| Typing indicator | Shows dots to say a reply is coming | No, it only signals that work is happening |
| Total response time | Time until the final word | Unchanged by streaming |
Our explanation, not a standard: streaming and the indicator are complementary. Show the dots during the gap before the first fragment, then replace them with streamed text.
Trade-offs to know about
Moderation is harder. OpenAI's guide notes that streaming partial output makes it harder to moderate completions, because partial text can be hard to evaluate, and that moderation scores requested with a generation arrive only after the full output exists. If your guardrails check the finished answer, a streamed customer may already have read it. Options include checking the question before generation, holding back risky topics from streaming, or accepting the risk for low-stakes content.
Errors arrive midway. A non-streamed call either works or fails. A streamed one can fail after half a sentence is on screen, so the widget needs a plan: show a retry line, or replace the fragment with a safe fallback message.
Intermediaries can swallow the stream. A proxy or content delivery layer that buffers responses will hold the pieces and release them all at once, which defeats the purpose. This is worth testing on the real production path, not only on a laptop.
Structure is harder to use half-finished. A reply in a strict machine-readable format cannot be parsed until it is complete, so streaming suits free text more than structured data.
FAQ
What is response streaming in a chatbot?
Sending the model's answer to the customer in small pieces as it is generated, so text starts appearing before the whole reply is finished.
Does streaming make the chatbot faster?
It makes the first words appear sooner, so the wait feels shorter. The time to the last word is about the same, since the model writes at the same speed.
Is streaming the same as a typing indicator?
No. A typing indicator is a placeholder animation. Streaming shows the actual answer text as it is produced.
What are server-sent events?
A browser feature where the server holds one HTTP connection open and pushes text messages to the page, one way only. Many chat streams use it.
Can I stream and still moderate answers?
Yes, but not by checking the finished reply alone. Check the question first, limit streaming for sensitive topics, or add a fallback for when a late check fails.
Does streaming cost more?
Providers generally bill by tokens, and streaming does not change how many are generated, but check your provider's pricing page for any difference.
Related terms
- Time to first token — the delay streaming is meant to shorten.
- Typing indicator — the placeholder shown before the first fragment.
- LLM token — the unit each fragment is made of.
- Chatbot first response time — the support-desk metric, which streaming can improve.
- AI guardrails — the checks that become harder when text is already on screen.
- Webhook — a different push pattern: one message to a URL, not an open stream.
Sources
- OpenAI, Streaming API responses — developers.openai.com/api/docs/guides/streaming-responses, read 20 October 2026. Source of
stream: true, delivery over server-sent events, the event namesresponse.created,response.output_text.delta,response.completedanderror, and the moderation caveat. - MDN, Using server-sent events — developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events, read 20 October 2026. Source of one-way transport,
text/event-stream, the message format, default reconnection and the six-connection limit without HTTP/2. - Chatbotscape evaluation methodology. /methodology (continuously updated).