Time to First Token — The Pause Before a Chatbot Starts Answering

Quick answer: Time to first token (TTFT) is the delay between sending a question to a large language model and seeing the first word of its answer arrive. It only becomes the number your customer feels if the answer is streamed, meaning shown piece by piece as it is written. Without streaming, the customer waits for the whole answer and TTFT is hidden inside total response time. TTFT is mostly a function of what happens before the model starts writing: your own retrieval and tool steps, the length of the prompt, the size of the model and how busy the provider is.

Three latency numbers that get mixed up

Total time is roughly TTFT plus the answer length divided by generation speed. That is why two models can swap places depending on the question: a model with a slow start and fast writing can beat a quick starter on a long answer, and lose on a one-line answer. Providers and benchmark sites use slightly different names for the second measure (tokens per second, inter-token latency, time per output token), so read the definition before comparing figures.

What goes into the wait

The following breakdown is our explanation of the parts, not a vendor specification. In a support chatbot the clock usually runs through these steps in order:

1. Your own work before the model call. Looking up the customer, searching the knowledge base in a retrieval-augmented setup, and calling tools all happen first. If the chatbot measures from the customer's message, this is often the largest piece and the one you control most.
2. The network and any gateway. Distance to the provider and any proxy or LLM gateway in between add a little each.
3. Queueing at the provider. At busy times a request can wait before processing starts.
4. Reading the prompt. The model has to process the whole input, including the system prompt, history and retrieved passages, before it can write the first word. Longer inputs generally mean a later first token.
5. Writing the first token. Larger models are generally slower per step than smaller ones.

OpenAI's latency guide makes a useful distinction here: it says generating output tokens is usually the highest-latency step, and gives as a rough heuristic that cutting output tokens by half may cut latency by about half, while cutting the prompt by half may improve latency by only about 1 to 5 percent. That is about total time. For TTFT specifically, the input still matters because it is processed before the first token, so very large contexts do push the first token later. Treat the guide's numbers as its rule of thumb, not a measurement of your system.

Why streaming changes how the wait feels

When a model call is not streamed, the server sends the full answer in one reply, and the customer sees nothing until it is complete. With streaming, OpenAI's guide explains, the server sends the output in small pieces as server-sent events so the application can start showing text while the model is still writing. The same guide describes streaming as the most effective approach for perceived latency, and notes that it can also shorten the overall wait for the reader because they start reading sooner.

So for a streamed chatbot, TTFT is the headline latency number: it is the time until the customer sees movement. For a non-streamed chatbot, total response time is. A typing indicator can cover a short gap, but it does not replace actual text.

One caveat from the same streaming guide: because partial output reaches the user before it is complete, it is harder to check content before it is shown, which can affect how you screen answers. If your guardrails need to see the full answer first, streaming and safety checks have to be designed together.

How long is too long?

There is no single good TTFT, and we do not quote one, because it depends on the model, region, prompt size and load, and published numbers change quickly. A general yardstick from outside AI comes from web.dev's RAIL article, which was written about web page interaction rather than chatbots: responses under about 100 milliseconds feel immediate, delays up to a second still feel like a continuous flow of thought, users start losing focus after a second, and past about 10 seconds they tend to become frustrated and may leave. The article also points readers to Core Web Vitals as its recommended way to set performance goals today. Use those thresholds as a reminder of how human patience works, not as a target for a model call.

The practical approach is to measure your own: set a ceiling for the share of conversations where the first text appears within a few seconds, track it weekly, and decide whether a miss is a retrieval problem, a prompt problem or a model problem.

What to log

• TTFT per request, measured from the moment your application sends the model request, and separately from the moment the customer's message arrived, so the gap shows your own pre-model work.
• Total response time and output length, because a slow answer may just be a long one.
• The model, the prompt size and whether retrieval or tools ran, so you can split the numbers into causes.
• A percentile, not just an average. A handful of very slow requests are what customers complain about, and an average hides them.

An LLM observability tool or your own logs can capture all of this. The production monitoring guide shows where it fits in a routine.

FAQ

What is time to first token?

The time between sending a request to a language model and receiving the first token of the response. It measures the wait before the answer starts appearing.

Is TTFT the same as latency?

No. Latency is a general term, and in practice most people mean total response time. TTFT is only the first part of it. Generation speed covers how fast the rest arrives.

What affects time to first token?

Your own retrieval and tool steps before the call, network distance and any gateway, provider queueing, the length of the prompt and the size of the model. The first of these is the one you control most.

Does streaming reduce TTFT?

Not by itself. Streaming does not make the model start sooner; it lets the customer see the first part as soon as it exists instead of waiting for the full answer. That makes the wait feel shorter.

Is a low TTFT always better?

Not if it costs answer quality. A smaller or faster model may start sooner but answer worse. Compare quality first, then speed, using a side-by-side test like the one in the smaller-model test guide.

Does prompt caching help?

It can. Prompt caching lets a provider reuse work on an unchanged prompt prefix, and OpenAI's guide advises putting the static part of the prompt first so caching works well. Check your provider's documentation for what it promises about speed and cost.

Related terms

• Chatbot first response time — the customer-facing support metric that this measurement feeds into.
• LLM token — the unit being counted, and what "first token" refers to.
• Prompt caching — a provider feature that can reduce the work done on a repeated prompt.
• Semantic caching — skips the model call for repeated questions, which removes the wait entirely on a hit.
• LLM observability — the logging that makes these numbers visible.
• Typing indicator — the visual cue that covers a short pause.

Sources

• OpenAI, Latency optimization — developers.openai.com/api/docs/guides/latency-optimization, read 14 October 2026. Source of the seven principles, the output-versus-input-token heuristic, the statement on streaming and perceived latency, and the advice to put static content first for caching.
• OpenAI, Streaming API responses — developers.openai.com/api/docs/guides/streaming-responses, read 14 October 2026. Source of how streaming works over server-sent events and the moderation caveat.
• web.dev, Measure performance with the RAIL model — web.dev/articles/rail, read 14 October 2026. Source of the response-time perception thresholds, which were written for web page interaction.
• Chatbotscape evaluation methodology. /methodology (continuously updated).