Query Rewriting· RAG pipeline
Query Rewriting — Fixing the Search Before the Search
Quick answer: A customer types "what about the annual one?" The words mean something in the conversation, but the knowledge base search sees only those five words and has nothing to match. Query rewriting uses a language model, or a few rules, to turn the message into something a search can use, such as "annual plan refund policy", before retrieval runs. It is cheap compared with fixing the documents, and it is one of the first things to try when a RAG chatbot fails on follow-ups and short messages.
Why the raw message is often a poor search query
Retrieval compares the query with passages, usually by meaning through vector embeddings in a semantic search index. That works best when the query is a complete, specific statement of what the customer wants. Real chat messages are often not:
- Follow-ups lean on earlier turns ("and for the annual one?", "does that include add-ons?").
- Short or vague messages carry too little meaning to land near the right passage ("pricing", "it broke").
- Customer wording differs from document wording, so the nearest passage by vocabulary is not the right one.
- Several asks in one message pull the search toward a blend that matches none of them well.
A weak query returns the wrong passages, and the model then answers from the wrong text or from nothing, which raises the risk of AI hallucination. Rewriting attacks the problem at the query rather than at the documents or the model.
The main kinds of rewriting
| Kind | What it does | Typical use | Cost |
|---|---|---|---|
| Standalone (condensed) question | Combines the latest message with the conversation so far into one self-contained question | Follow-ups in multi-turn chat | One short model call, only when there is history |
| Query expansion | Adds synonyms or related terms to the query | Vocabulary mismatch, short queries | Rules or one model call |
| Multi-query | Generates several differently worded versions, searches each and merges the results | Ambiguous or broad questions | One model call plus several searches |
| Decomposition | Splits a message with several asks into one query per ask | "Can I change plan mid-cycle and does the refund apply to add-ons?" | One planner call plus several searches |
| HyDE | Has a model write a hypothetical answer, then searches with that text instead of the question | Questions phrased very differently from the documents | One generation call, longer text to embed |
| Trained rewriter | A small model trained to write queries that help the answering model | Teams with data and engineering time | Training effort |
Most small support bots need only the first row. The rest are tools for specific failures that a test has shown.
What the sources say
These are statements from the sources, not our test results.
- LlamaIndex. The reference for its condense question chat engine describes the flow as "First generate a standalone question from conversation context and last message, then query the query engine for a response." In the code the chat history and latest message are sent to the model with a condense prompt, and when there is no history the latest message is used as it is. That second detail matters: a sensible build skips rewriting for the first turn.
- HyDE. The paper "Precise Zero-Shot Dense Retrieval without Relevance Labels" (Gao, Ma, Lin and Callan, arXiv 2212.10496, December 2022) describes Hypothetical Document Embeddings. An instruction-following model writes a hypothetical document for the query, which may contain made-up details, and an encoder turns it into a vector used to find real documents. The authors argue the encoder filters out the incorrect details. The abstract reports results against an unsupervised retriever and comparable results to fine-tuned retrievers on the tasks they studied; that is research on benchmark tasks, not on a support chatbot.
- Rewrite-Retrieve-Read. The paper "Query Rewriting for Retrieval-Augmented Large Language Models" (Ma, Gong, He, Zhao and Duan, arXiv 2305.14283, 2023) adds a rewriting step before retrieval and trains a small model as the rewriter using feedback from the answering model. The authors report consistent gains on question-answering tasks, with a web search engine as the retriever.
The three differ in ambition: LlamaIndex shows the plain, practical follow-up fix, HyDE changes what is embedded, and the third trains the rewriter. Only the first is a one-hour job.
Where rewriting goes wrong
- It can drop or change a detail. A rewrite that turns "does the Pro plan include SMS?" into "does the plan include messaging?" has lost the plan name, and the search now returns the wrong page.
- It can add things the customer never said. Expansion and HyDE introduce words and invented details. They help matching, but a model asked to rewrite can also assume a plan, a country or an order that is not in the conversation.
- It adds delay and cost. Each rewrite is another model call before the search, which lengthens time to first token. Skip it when it cannot help, such as on the first turn.
- It hides the original wording. If you log only the rewritten query, you cannot see what the customer actually typed. Keep both.
- It does not fix a bad index. If the answer is missing, badly chunked or buried under near-duplicates, a better query cannot find it. A reranker is a different fix for a different failure.
- It needs its own test. Judge rewriting on whether the right passage is retrieved and whether answers improve, using a golden dataset and the methods in LLM evaluation, with the rewrite switched on and off.
Rewriting compared with neighbouring ideas
Rewriting changes the query before retrieval. Reranking reorders the passages after retrieval. Agentic RAG lets the model decide, step by step, whether to rewrite, split, search again or stop; rewriting is the cheapest step inside it. Context engineering is the wider job of deciding what goes into the model's window, and the rewritten query is one input to that. Session context is the memory of the conversation that the standalone-question step reads from.
FAQ
What is query rewriting in RAG?
A step before retrieval that changes the customer's message into a better search query. The most common version turns a follow-up that depends on earlier turns into a self-contained question.
Why does a RAG chatbot fail on follow-up questions?
The search sees only the latest message. A phrase like "what about the annual one?" has no meaning without the earlier turns, so it matches the wrong passages or none. Rewriting adds the missing context.
What is the difference between query rewriting and query expansion?
Rewriting replaces or restates the query, for example as a standalone question. Expansion keeps the query and adds related terms. Both aim at a better match, and they can be combined.
What is HyDE?
Hypothetical Document Embeddings. A model writes a made-up answer to the question, and that text, rather than the question, is used to search. It can help when questions and documents use very different wording, and it adds a generation step.
Does query rewriting slow the chatbot down?
Yes, by one model call, so keep the prompt short, use a small fast model if it passes your test, and skip the step when there is no history.
How do I know rewriting is helping?
Run the same test questions with the rewrite on and off and compare whether the right passage is retrieved, then whether the final answers improve. The companion guide shows how.
Related terms
- Retrieval-augmented generation — the pipeline rewriting belongs to.
- Semantic search — the search that benefits from a clearer query.
- Reranking — the fix for a different retrieval failure, after the search.
- Agentic RAG — where rewriting is one of several model-driven steps.
- Vector embeddings — what the rewritten query is turned into.
- AI hallucination — the risk a poor retrieval raises.
- LLM evaluation — how a rewriting change is tested.
Sources
- Gao, Ma, Lin and Callan, Precise Zero-Shot Dense Retrieval without Relevance Labels — arxiv.org/abs/2212.10496, abstract read 19 October 2026.
- Ma, Gong, He, Zhao and Duan, Query Rewriting for Retrieval-Augmented Large Language Models — arxiv.org/abs/2305.14283, abstract read 19 October 2026.
- LlamaIndex, Condense Question chat engine reference — developers.llamaindex.ai, read 19 October 2026.
- Chatbotscape evaluation methodology. /methodology (continuously updated).