Skip to content
Chatbotscape
Editorial flat-vector illustration for How to Stream Chatbot Answers into a Website Widget (and Check That It Really Streams)
Published 8 min read

How to Stream Chatbot Answers into a Website Widget (and Check That It Really Streams)

Quick answer: Turn on streaming at the model provider, have your server forward each fragment to the browser as it arrives (server-sent events are the common choice), append fragments to the chat bubble, and plan for the two things streaming breaks: late errors and moderation. Then test through your real production path, because a proxy that buffers responses will quietly turn a stream back into one delayed block.

Step 1: Measure the wait before you change anything

Streaming is worth the extra code only if customers wait on long answers. Log, for 50 to 100 real conversations, two timestamps per answer: when the customer's message reached your server, and when the first and last parts of the answer reached the browser. The gap to the first is the figure explained in time to first token; the gap to the last is total response time.

If most answers are one or two sentences and finish in about a second, streaming gains little. If many run to several paragraphs, or retrieval and tool calls come before the model starts writing, it gains a lot. Note that streaming only shortens the wait for the first words, so a slow retrieval step before generation still has to be fixed on its own.

For context on why the first seconds matter: the RAIL performance model on web.dev says a result within 100 ms feels immediate, that beyond one second users lose focus on the task, and that beyond ten seconds they are frustrated and likely to abandon it. Those are general interface thresholds, not chatbot benchmarks.

Step 2: Switch streaming on at the provider

With OpenAI's Responses API, the guide says to set stream: true on the request, and the response then arrives as typed events delivered over server-sent events. For text, the events to handle are response.created, response.output_text.delta (the new piece of text), response.completed and error. Other providers use different event names, so read yours rather than copying these.

In pseudocode, the loop on your server is: for each event, if it is a text-delta, forward the fragment; if it is completed, send a final marker; if it is an error, send an error marker. Keep that mapping in one small function so a provider change touches one place.

Step 3: Relay the stream to the browser

Your server needs to answer the widget's request with a response that stays open and carries the fragments. Server-sent events fit well. Per MDN, the response uses the content type text/event-stream, each message is field: value lines ended by a blank line, and the connection is one-way from server to browser.

Two practical points. First, the browser's EventSource sends only GET requests, so if the question travels in a POST body, read the stream from a fetch call instead, or create the conversation first and stream from a URL that carries its identifier. Second, WebSockets also work and allow two-way messages, but for a chat where the customer's message goes up as an ordinary request and only the answer streams down, the simpler one-way stream is usually enough. That is our judgment, not a rule.

Send a short comment line every few seconds as a keep-alive during a long retrieval step. MDN notes that a line starting with a colon is treated as a comment, which is useful for this.

Step 4: Render fragments in the browser

Append each fragment to the open message bubble. Three habits prevent most visible problems:

  1. Insert as text, not as HTML. The model's output is untrusted. Add fragments as plain text, and convert Markdown to markup only after a sanitizer has cleaned it, or at the end.
  2. Show the typing dots until the first fragment. The typing indicator covers the gap before text starts; remove it when the first fragment lands.
  3. Keep the view scrolled sensibly. Auto-scroll only if the customer is already at the bottom, so you do not drag them away from text they are reading. Announce completion, not every fragment, to screen readers, as covered in the chatbot accessibility guide.

Step 5: Plan for errors and moderation

A streamed answer can fail after half a sentence is on screen. Decide the behavior in advance: keep the fragment and add a line such as "Something went wrong; please try again" with a retry button, or remove the fragment and show a fallback message. Also decide what the browser does when the connection drops: the default EventSource behavior is to reconnect, which could repeat an answer, so give each answer an identifier and ignore duplicates.

Moderation needs a decision too. OpenAI's guide notes that streaming makes completions harder to moderate because partial text can be difficult to evaluate, and that moderation scores requested with a generation arrive only after the full output exists. If your guardrails work on the finished reply, a streamed customer may have read it already. Options: check the incoming question first and refuse before generating; turn streaming off for sensitive topics such as billing disputes; or accept the risk for low-stakes answers and log everything for review. See the guardrails guide for the wider set of checks.

Step 6: Test through the real production path

This step catches the most common failure. Run the widget against the live site, not a local server, and watch whether text appears gradually or all at once at the end. If it arrives in one block, something between the model and the browser is buffering. Likely culprits include a reverse proxy, a content delivery layer, compression, or a platform setting that collects the full response first. Look in the documentation of each layer for a setting about response buffering, and try again.

Also test: a very long answer; closing the tab mid-answer (the server should stop generating, or you pay for tokens nobody reads, see estimating token usage and cost); several tabs open at once, since MDN notes browsers allow only six server-sent-event connections per domain without HTTP/2, shared across tabs; and a forced provider error.

Check the arithmetic

An invented example, not a measurement. Suppose a long answer takes 1.5 seconds before the model produces its first fragment, then 7.5 seconds more to write the rest.

  • Without streaming, the first text appears at 1.5 + 7.5 = 9.0 seconds.
  • With streaming, the first text appears at 1.5 seconds, and the last at 9.0 seconds as before.
  • The wait for first text falls by 9.0 − 1.5 = 7.5 seconds, which is 7.5 ÷ 9.0, about 83 percent.

Two honest readings. The total time is unchanged, so a customer who needs the final line still waits 9 seconds. And 1.5 seconds is still above the one-second mark in the RAIL guidance, so if that wait matters, the next lever is the pre-generation work (retrieval, prompt length, prompt caching), not more streaming.

Common mistakes

  • Testing only on a laptop. Local servers do not buffer; production often does.
  • Inserting model output as HTML. Treat it as untrusted text.
  • No plan for a mid-answer error. The customer is left with a sentence that stops.
  • Streaming where you must check first. For answers that need a complete check, such as prices or legal wording, show the finished reply.
  • Forgetting to stop generation when the customer leaves, which wastes tokens.

FAQ

Do I need WebSockets to stream a chatbot answer?

Not usually. A one-way server-sent event stream carries the answer down, and the customer's message can go up as an ordinary request. WebSockets help if you need continuous two-way messaging.

Why does my stream arrive all at once?

Something in the path is buffering the response, often a proxy, a delivery layer or compression. Test on the live site and check each layer for a buffering setting.

Does streaming reduce my model bill?

Not by itself. The same tokens are generated. It changes when the customer sees them. Check your provider's pricing for any difference.

Can I stream and still apply guardrails?

Yes, with limits. Check the question before generating, withhold streaming for sensitive topics, and have a fallback if a late check fails.

Should every answer stream?

No. Short answers gain little, and answers that need a full check before display are better shown whole.

Does a hosted chatbot platform stream?

Some do and some do not. Ask the vendor, then run the Step 6 test on a long answer.

Sources

About this guide

Chatbotscape launched in 2026 as an independent review site for chatbot platforms. This guide is editorial guidance built from the sources above; the measure-first step, the relay choices, the error and moderation options, the production test and the worked example are our own practice with invented numbers, and we have not benchmarked any provider or framework.

Last updated

21 October 2026 — first published.