How to Rewrite Follow-Up Questions for a RAG Chatbot (and Prove It Helps)

Quick answer: Find the follow-up questions your chatbot answers badly, add one short model call that turns the latest message plus the recent turns into a self-contained search query, skip that call on the first turn, log the original and the rewritten text, and then run the same test questions with the rewrite on and off. In our invented example, retrieval on follow-ups rises from 40 to 75 percent and the extra call costs about $1.60 a month, but the test also shows where a rewrite loses a detail.

Step 1: Confirm that follow-ups are your problem

Rewriting fixes one failure: the search cannot find the right passage because the last message depends on earlier turns or is too short to match anything. It does nothing for a document that is missing or badly split.

Take 30 to 50 conversations where the bot gave a wrong or empty answer on turn two or later. For each, read the retrieved passages and write one line: was the right passage missing because the message alone was unclear ("and the annual one?"), or because the passage is not in the knowledge base at all? Count the first kind. If fewer than a fifth of the failures are of that kind, fix the content or the search itself first. This sorting is our own practice and takes about an hour.

Step 2: Build a small test set before changing anything

You need a fixed set to compare against. Write 100 questions with the passage that should be retrieved for each, and make sure at least 30 of them are follow-ups, each stored with the earlier turns that give it meaning. Include a few follow-ups whose first message contains a plan name, a product name or a date, because those are the details a rewrite tends to lose. This is a small golden dataset. Run the current bot on it and record, for each question, whether the right passage appears in the top five results. That count is your baseline. A set of 100 is a smoke test: differences of a few questions are noise, so look for large, consistent changes.

Step 3: Write the rewrite prompt

The rewrite is one model call with a short prompt. A pattern that works as a starting point:

1. Give the recent turns, the last four to six messages rather than the whole conversation, and the latest message.
2. Ask for one self-contained question or search query that a person who has not seen the conversation would understand.
3. Say what to keep: product names, plan names, numbers, dates and quoted error messages, exactly as written.
4. Say what not to do: do not answer, do not add details that are not in the conversation, and if the latest message is already self-contained, return it unchanged.
5. Ask for the query only, with no explanation, so the output can be passed straight to the search.

The LlamaIndex documentation describes the same idea for its condense question chat engine: first generate a standalone question from the conversation context and the last message, then query. The prompt wording above is ours, and you should expect to edit it after the first test.

Use the smallest model that passes your test in Step 5. A rewrite is a short, constrained task, which is why it is a natural job for a smaller model.

Step 4: Skip the first turn, and log both versions

Two rules keep the step cheap and safe.

Skip it when there is no history. On the first message there is nothing to add, so send the text straight to the search. The LlamaIndex code does the same: with no chat history the latest message is used as it is. This alone removes the call from every conversation's opening turn.

Log the original and the rewrite side by side. If you store only the rewritten query, you cannot tell later whether the customer said something odd or the rewriter did. Keep both, and show the rewritten text to your reviewers.

Also decide what the answering model sees. Usually it should still get the customer's real message and the recent turns, with the passages found by the rewritten query. The rewrite is for the search, not a replacement for the conversation. The context window guide covers how much history to keep.

Step 5: Test with the rewrite on and off

Run the same 100 questions twice, once with the rewrite and once without, and compare.

1. Retrieval hit rate. For each question, does the right passage appear in the top five? Report it separately for follow-ups and for stand-alone questions. The follow-up figure should rise. The stand-alone figure should not fall.
2. Damaged rewrites. Read every rewrite on the follow-ups and mark the ones that lost or changed a detail. Count them. These are the cost of the method.
3. Answer quality. Pick the questions where the on/off result differs, and have a person judge the final answers. For the rest, a model-as-judge check that a person samples is enough. The RAG evaluation guide explains how to score retrieval and answers separately.
4. Delay. Measure the extra time before the first word of the answer, with the median and the slowest tenth. The guide to faster replies shows where else to save it.
5. Pass bar, set before you look. For example: follow-up hit rate up by at least 20 points, stand-alone hit rate down by no more than 2, and no more than 5 percent of rewrites damaged.

If the bar is not met, change the prompt and test again before trying another method. Expansion, multi-query and HyDE all add calls or searches, so earn them with a failure the simple step could not fix.

Step 6: Release gradually and watch

Turn the rewrite on for a share of live traffic first, for example 10 percent. Compare that group with the rest on the signals you already track, such as the fallback rate, handoffs to a person and customer ratings. Review a weekly sample of original and rewritten pairs, because customers phrase things differently from your test set, and re-run the test after any change to the model, the prompt or the knowledge base.

Check the arithmetic

Every number below is invented for illustration, not a measurement or a vendor price.

Three lessons. First, the gain is concentrated: 21 more follow-ups are found, and the overall rate rises 7 points because only a fifth of the questions are follow-ups. Second, skipping the first turn is what keeps the monthly cost at $1.60 rather than $8.00, which shows that the rule is worth more than the choice of model. Third, 15 follow-ups still fail, and 3 of the 60 rewrites lost a detail, so read the failures: they are your next improvement, whether that is a better prompt, reranking or a missing article. The decision rule we use: keep the rewrite if the bar from Step 5 is met on your own questions.

Common mistakes

• Rewriting every turn. It costs more and slows the first reply, for no gain.
• Letting the rewriter answer or guess. Tell it to return the query only and never to add a plan, country or order that was not mentioned.
• Passing the whole conversation. Long histories add cost and noise. Use the last few turns.
• Logging only the rewrite. You lose the evidence for every later complaint.
• Judging by a few examples. Look at hit rates over a fixed set, plus every damaged rewrite.
• Expecting it to fix missing content. If the answer is not in the knowledge base, no query will find it.

FAQ

What is the standalone-question step in a RAG chatbot?

One model call that restates the latest message, using the recent turns, as a question that makes sense on its own. It is then used for the knowledge base search.

Should I rewrite every message?

No. Skip the first turn and any message that is already self-contained. In our invented example, that keeps the cost at $1.60 a month instead of $8.00.

Does query rewriting slow the chatbot down?

Yes, by one model call on the turns where it runs. Measure it on your own setup, keep the prompt short, and use the smallest model that passes your test.

Can the rewrite make answers worse?

Yes. A rewrite can drop a plan name or add an assumption, which sends the search to the wrong page. That is why the test counts damaged rewrites and the log keeps the original text.

Do I need HyDE or multi-query as well?

Only if a test shows a failure the simple rewrite cannot fix. Both add calls, and the query rewriting entry explains what each does.

How do I measure whether it helped?

Compare retrieval hit rates on a fixed question set with the rewrite on and off, separately for follow-ups and stand-alone questions, then check answer quality on the questions that changed.

Related guides

• Agentic RAG chatbot guide — rewriting as the first of four steps.
• Hybrid search and reranking guide — for wrong-document failures that are not follow-ups.
• RAG evaluation guide — scoring retrieval and answers.
• Build a RAG chatbot — the pipeline this step sits in.
• Query rewriting, retrieval-augmented generation and semantic search — the glossary entries behind the terms used here.

Sources

• LlamaIndex, Condense Question chat engine reference — developers.llamaindex.ai, read 19 October 2026: standalone question from conversation context and last message; latest message used as is when there is no history.
• Gao, Ma, Lin and Callan, Precise Zero-Shot Dense Retrieval without Relevance Labels — arxiv.org/abs/2212.10496, abstract read 19 October 2026 (HyDE, mentioned only).
• Ma, Gong, He, Zhao and Duan, Query Rewriting for Retrieval-Augmented Large Language Models — arxiv.org/abs/2305.14283, abstract read 19 October 2026 (rewriting before retrieval, mentioned only).
• Chatbotscape evaluation methodology. /methodology (continuously updated).

About this guide

Chatbotscape launched in 2026 as an independent review site for chatbot platforms. This guide is editorial guidance built from the sources above; the failure-sorting step, the prompt pattern, the skip-and-log rules, the on/off test and the worked example are our own practice with invented numbers, and we have not benchmarked any rewriting method.

Last updated

20 October 2026 — first published.