Does Your Chatbot Need GraphRAG? A Five-Step Test Before You Build

Quick answer: Do not decide from the hype. Sort a sample of your real questions, run them through the plain retrieval setup you already have, and look only at the ones it gets wrong. If most failures are about connections between facts or about the whole collection, try a graph on those; if they are about missing, outdated or badly split articles, fix those first. In our invented example a graph build repays itself in about five and a half months, but only because 8 percent of questions were relationship questions that plain retrieval often missed.

Step 1: Make sure plain retrieval is already good

GraphRAG is an upgrade for a retrieval setup that works, not a repair for one that does not. Before anything else, check the cheaper levers: are the knowledge base articles current and self-contained, are chunks split sensibly, would hybrid search or a reranker help, and have you measured retrieval quality? Many failures blamed on "vector search can't do relationships" are really a missing article or a chunk that cut a table in half. If you skip this step, you will pay for a graph and keep the same bug.

Step 2: Sort your questions by what they need

Take 100 recent customer messages, or 100 realistic test questions. Sort each by what a correct answer requires:

• Lookup: one fact from one place. "What is the return window?"
• Connection: two or more facts that must be joined. "Does the Pro plan include the integration I need, and does that integration support my region?"
• Whole-collection: a summary across many documents. "What are the most common changes in this year's policy updates?"
• Out of scope: the answer is not in your documents at all. These should go to a person or a polite refusal, whatever the retrieval design.

Count each pile. This sorting is our own practice and takes about an hour. The published case for GraphRAG is about the last two piles. The research paper describes its gains on global questions over large collections, where plain retrieval struggles because no single chunk holds the answer. If your connection and whole-collection piles together are a few percent of traffic, the case is weak. A support chatbot is mostly lookups.

Step 3: Build a test set and score plain RAG

For the connection and whole-collection piles, write at least 30 questions with a correct answer or a checklist of facts that a correct answer must contain. Add 20 lookup questions as a control. Build a small golden dataset if you have none. Run all 50 through your current retrieval setup and score each answer: correct, partly correct, wrong or invented. A person should score the connection and whole-collection answers; a model-as-judge check on a sample can handle the rest. Record the failure rate for each pile. If plain retrieval already answers 90 percent of the connection questions correctly, stop here: you do not need a graph.

Step 4: Try the graph on a small slice

If plain retrieval fails often on connection or whole-collection questions, build a graph on a small slice, not the entire knowledge base. The Microsoft GraphRAG README warns that indexing can be expensive and recommends starting small, so index perhaps the 30 to 50 documents that the failing questions depend on. Pick one tool from the GraphRAG entry: Microsoft's pipeline if you need whole-collection answers, LightRAG if lighter updates matter, or Neo4j or LlamaIndex if you want building blocks. Each claim in a project's documentation is the project's own, so do not carry its benchmark numbers into your plan.

Practical choices to make:

1. Standard or fast indexing. Microsoft's documentation says the standard method uses a language model for every reasoning step and that extraction is roughly 75 percent of indexing cost, while the fast method is much cheaper but gives a noisier graph and suits mainly summary questions. Try the one that matches your pile.
2. Which search mode. Local search for entity questions, global search for whole-collection questions, according to the documentation. Global search is described as resource-intensive, so estimate its cost per question.
3. How updates work. Ask how a changed article reaches the graph and the summaries, and how long that takes, since stale relationships create confident wrong answers.

Whichever approach you use, make the graph an addition: retrieve from the graph and the text, and keep plain retrieval as the default for lookups. A question-type check that sends only connection and whole-collection questions to the graph keeps the cost down, much like the safe default in routing questions to a cheaper model.

Step 5: Compare the scores and apply a decision rule

Run the same 50 questions through the graph setup and score them the same way. Set the pass bar before you look at results. Our rule of thumb, which is our own and not a vendor method:

• Adopt the graph for the piles where it fixes at least a third of the failures, and the saving covers the build in a period you accept.
• Keep plain retrieval for lookups, and for any pile where the graph shows no clear gain.
• Check that the graph did not make lookups worse and did not add invented facts.
• Ask a second person to score a sample blind, because the people who built the graph tend to see what they hope to see.

Fifty questions is a smoke test, not a proof; a small difference at this size may be noise. If the call is close, collect more questions, or run the new setup beside the old one on a share of live traffic and compare escalation to a person, as described in the production monitoring guide. Keep the test set for later: rerunning it after each change is the idea behind the regression testing guide.

Check the arithmetic

Every number below is invented for illustration, not a measurement or a vendor price.

Three lessons. First, in this example the model charges for indexing a small knowledge base are tiny; the real cost is the people hours, the extra system to maintain and the questions that go wrong when the graph is stale. Second, the result depends almost entirely on the relationship share: at 2 percent instead of 8 percent, the saving falls to $48 and the net saving is about $42 and the payback stretches to about 24 months, which is why Step 2 comes first. Third, a larger corpus, a standard rather than fast index or frequent global searches multiply the model cost, so rerun the table with your own volumes and the token cost estimate. The rule we use: if you cannot show failures on a measured set, do not build.

Common mistakes

• Building a graph to fix a retrieval bug. Check chunks, articles and search settings first.
• Indexing everything at once. Start with the documents the failing questions need.
• Trusting published gains. The reported results are on the authors' datasets, not on yours.
• Ignoring updates. A graph that lags your policy changes is worse than none.
• Scoring by the builders. Have someone else check a sample.
• Sending lookups through the graph. It adds cost and delay for no gain.

FAQ

When should a chatbot use GraphRAG instead of vector search?

When a measurable share of real questions need facts joined across documents or a summary across the whole collection, and plain retrieval fails on them in your own test. For single-fact lookups, vector search with good chunks is usually enough.

Is GraphRAG overkill for a small knowledge base?

Often, yes. A few hundred short, self-contained articles rarely need it. Step 3 tells you for your own case, and if plain retrieval answers most connection questions correctly, skip the graph.

How much does GraphRAG cost compared with standard RAG?

Microsoft's documentation says extraction is about 75 percent of indexing cost and that the fast method is much cheaper. In our invented example the model charges are small and the build hours dominate. Run the table with your own numbers.

How do I test GraphRAG against vector RAG?

Build a set of connection, whole-collection and lookup questions with known answers, score both setups the same way before and after, set the pass bar first, and have someone else score a sample.

Which tool should I start with?

Choose by the question type. The documentation describes Microsoft GraphRAG for whole-collection questions, LightRAG as a lighter option with incremental updates, and Neo4j and LlamaIndex as building blocks. We have not benchmarked any of them.

Does GraphRAG stop hallucinations?

No. The model can still misread or invent, so keep answer checks, and see the hallucination guide.

Related guides

• Build a RAG chatbot — the plain retrieval baseline.
• Hybrid search for RAG — a cheaper lever to try first.
• Agentic RAG for a chatbot — a different way to improve multi-part questions.
• Measure RAG quality — the scoring behind Steps 3 and 5.
• GraphRAG, retrieval-augmented generation and agentic RAG — the glossary entries behind the terms used here.

Sources

• Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization — arxiv.org/abs/2404.16130, read 18 October 2026 (abstract).
• Microsoft, GraphRAG documentation, indexing methods and query overview — microsoft.github.io/graphrag, read 18 October 2026.
• Microsoft, GraphRAG, GitHub repository — github.com/microsoft/graphrag, read 18 October 2026: indexing cost warning.
• HKUDS, LightRAG, GitHub repository — github.com/HKUDS/LightRAG, read 18 October 2026.
• Chatbotscape evaluation methodology. /methodology (continuously updated).

About this guide

Chatbotscape launched in 2026 as an independent review site for chatbot platforms. This guide is editorial guidance built from project documentation; the question-sorting step, the failures-first test, the decision rule and the worked example are our own practice with invented numbers, and we have not run any graph-based retrieval tool.

Last updated

19 October 2026 — first published.