Build a RAG Chatbot — Three Routes, the Settings That Matter, and When to Skip It (2026)
Quick answer: A RAG chatbot answers from your documents instead of from the model's memory, and there are three ways to get one: a platform where retrieval is built in (an afternoon of work), a builder with knowledge-base nodes wired into your own flows (days), or a custom pipeline you assemble from an embedding model, a vector database, and an LLM (weeks, plus an engineer who stays). The tutorials that phrase attracts mostly teach route three. Most SMBs should take route one or two, spend the saved time on the knowledge base itself, and revisit custom infrastructure only when a platform limit actually bites. The build is the easy part to buy; the content and the testing are the parts nobody can sell you.
The pull toward over-building is real because the custom stack is what the tutorials teach and what the infrastructure vendors market. This guide takes the opposite path: it starts from the decision, walks the platform route in detail because that is the one most readers should take, and is specific about the handful of situations where custom work earns its keep.
Pick your route before your tools
The fork depends on two questions, not on ambition: where do your answers need to appear, and does anything about your data forbid a third-party platform?
Route one — a platform with retrieval built in. Products such as Chatbase exist almost entirely for this job: upload documents or point at URLs, get a bot that answers from them with sources attached. Support suites and site-widget products increasingly bundle the same capability under names like AI training or knowledge answers. If your goal is a website or help-center bot that answers product questions, this route gets you to a testable bot in hours, and the glossary entry on RAG explains everything happening under the hood.
Route two — a builder with knowledge nodes. Platforms such as Botpress and Voiceflow treat retrieval as one node type inside a flow you design. Take this route when the RAG answer is one behavior among several (order lookups, structured lead capture, channel routing, human handoff) and you want to control exactly when the bot consults documents versus follows a scripted path. More control, more decisions, more testing surface.
Route three — the custom pipeline. Embedding model, vector store, retrieval logic, prompt assembly, and hosting, all yours. Legitimate reasons exist: data that cannot leave your infrastructure, a bring-your-own-LLM requirement with unusual constraints, retrieval over millions of records, or RAG embedded in a product rather than bolted onto support. If none of those describes you, route three is a hobby wearing a business case. Teams already comfortable shipping code can weigh the trade-offs in our builder-versus-vibecoding guide.
The build, in the order that matters
What follows assumes route one or two. The sequence holds for custom builds too, with more work at every step.
Start with the content, not the connector. Retrieval searches whatever you feed it, and thin or contradictory content is the leading cause of wrong answers, ahead of model choice and ahead of settings. Before uploading anything, run the question-first audit from our knowledge-base guide: list the questions customers actually ask, check that a written answer exists for each, and fix the gaps in the source material first. The knowledge base is the input; RAG is just the delivery.
Connect sources the boring way. Prefer the platform's native connectors (help center, sitemap, file upload) over copy-paste, because native connections can re-sync when content changes and pasted text silently rots. Note which sync is automatic and which needs a manual refresh: a stale index answering from last quarter's pricing page is one of the most common RAG failures in the wild, and the fix is a calendar reminder, not an engineer.
Scope what the bot may answer from. Everything you index is something the bot can say. Internal wikis, draft policies, and old promotions do not belong in a customer-facing index. If the platform supports collections or source groups, separate customer-safe content now; if it does not, curate before upload. Treat anything with personal data under the rules in our security and PII guide.
The settings that matter more than the model
Platforms compete on model names, but four less-advertised controls decide most of the observed quality:
Grounding strictness. Somewhere in the settings is an instruction governing whether the bot may answer beyond retrieved passages. Set it strict: answer only from the provided sources. A creative fallback to the model's general knowledge feels helpful in a demo and produces confident nonsense about your actual product in production, the failure users report as "the bot lied."
The refusal path. Decide what happens when retrieval comes back empty or weak. The good pattern is a plain "I don't have that in my docs" followed by an offer to reach a person; the bad pattern is answering anyway. This is the same guess-ask-or-escalate trade our confidence-policy guide covers for intent bots; generative bots need the policy just as much and hide it more.
Citations on. If the platform can show which sources fed an answer, enable it. Users trust verifiable answers more, and your team debugs a wrong answer in seconds by seeing which passage misled the model.
Retrieval breadth, if exposed. Some platforms let you set how many chunks are retrieved per question. More is not better: every extra chunk spends context-window budget and adds material the model may blend into the answer. If answers feel diluted or mix unrelated topics, and the dial exists, try turning it down before switching models; for conversation-length problems, see our context-window management guide.
Test like a skeptic before launch
A RAG bot demos well by construction, which is exactly why the demo proves nothing. Before launch, build a written question set of thirty to fifty entries drawn from real tickets and chat logs, and make sure it includes the uncomfortable cases: questions whose answers are not in the knowledge base (the correct behavior is refusal), questions containing exact identifiers such as SKUs and error codes (semantic search's known blind spot, per the vector-database entry), paraphrases that share no words with the source article, and near-duplicate topics whose answers differ (shipping domestic versus international). Score answers on a three-way scale: correct, wrong, or correctly declined. A wrong answer is worse than a refusal, and a test set that contains no unanswerable questions cannot tell you the difference. Fold the set into the regression pass in our QA testing protocol so it runs again after every knowledge-base change, not just once at launch.
What to check after 30 days
Four checks, in the order they usually pay off. First, read a sample of transcripts where the bot answered from documents (twenty is enough to start) and grade them with the same three-way scale; wrong-but-confident answers cluster around the same few source passages, and rewriting those passages fixes whole categories at once. Second, mine the refusals: every "I don't have that" is either a missing article (write it) or a retrieval miss on existing content (usually a chunking or phrasing problem; restructure per the knowledge-base guide). Third, watch the fallback rate trend line as the closest thing a generative bot has to a single health metric, using the playbook from our fallback-reduction guide. Fourth, check the invoice: RAG spends tokens on every retrieval-backed turn, and pricing models differ enough across platforms that the cost-per-resolved-conversation math from our pricing guide is worth re-running on real volume rather than the trial month.
When to skip RAG entirely
Not every bot needs retrieval, and wiring it in adds cost and a new failure surface. Skip it when the bot's job is flows, not answers: welcome sequences, abandoned-cart nudges, lead capture, booking. Skip it when the honest source material is a dozen FAQs; a well-built menu bot answers those with zero hallucination risk and no per-token cost. And skip the whole project when conversation volume is too low to justify maintaining an index; the arguments in when not to use a chatbot apply with extra force to the AI-equipped kind, because an unmaintained RAG bot degrades more convincingly than a broken menu. Platform-fit questions across these categories are ranked in best AI chatbot, scored under our published methodology.
Frequently asked questions
What do I need to build a RAG chatbot?
On a platform: a curated set of documents and an afternoon. The platform supplies the embedding model, the vector database, and the LLM. On a custom build: those three components plus chunking logic, prompt assembly, hosting, and an owner for each. The component everyone actually lacks is a knowledge base written to be retrieved. Start there.
Do I need to know how to code?
Not for routes one and two. Retrieval-equipped platforms and builder knowledge nodes are configuration, not programming. Code enters with route three, or when you want retrieval behavior a platform does not expose.
Which vector database should I use?
If you are on a platform, none — it is built in and not your decision. If you are building custom at SMB document scale, the unexciting answer is that the database choice is rarely what limits quality; content and chunking are. Adding vectors to a database you already operate (pgvector on PostgreSQL is the common pattern) avoids running a new system; the glossary entry maps the landscape without the vendor noise.
Why does my RAG bot still make things up?
Three usual suspects, in order: the grounding instruction permits answering beyond retrieved passages (tighten it), retrieval returned plausible-but-wrong chunks the model paraphrased (fix the content or its chunking), or the answer plain is not in the knowledge base and the bot guessed instead of refusing (fix the refusal path). Grounding reduces hallucination; nothing eliminates it — which is why the test set needs unanswerable questions.
How much does a RAG chatbot cost to run?
Platform plans typically bundle retrieval into AI-message or credit pricing; custom builds pay for embedding, vector-store hosting, and LLM tokens separately. Published per-query figures age fast, so we keep them out of evergreen guides: run your expected volume through the three pricing models in our pricing guide and price the custom route honestly, including the engineer.
RAG or fine-tuning for a support bot?
RAG, almost always: it updates when your docs update and can cite sources. Fine-tuning changes a model's style and domain fluency, not its access to current facts. The glossary entry covers the comparison; for support content that changes weekly, retrieval is the only maintainable option.
Related guides
- Retrieval-augmented generation (glossary) — the architecture this guide assumes; definition, diagram, and limitations
- Vector database (glossary) — the search layer inside every route, and when it becomes your decision
- Build a chatbot knowledge base — the content work that determines answer quality before any setting does
- Chatbot confidence policy — guess, ask, or hand off; the refusal design RAG bots hide
- Chatbot QA testing protocol — the regression discipline the launch test set should join
- Reduce chatbot fallback rate — reading the closest thing RAG has to a health metric
- Pricing AI chatbots: three models — how retrieval-backed turns show up on the invoice
- Chatbot builder vs vibecoding — the build-it-yourself trade, examined beyond RAG
- Best AI chatbot — retrieval-capable platforms, ranked under our published methodology
About this guide
Chatbotscape launched in 2026 as an independent review site for chatbot platforms. This guide is part of our SMB chatbot Academy. It is editorial guidance: the route framework and configuration advice are working practices for retrieval-backed bots, not guarantees, and the skip-the-custom-build advice runs against the AI-infrastructure industry's commercial interest deliberately. Your data constraints, volume, and engineering capacity decide the arithmetic. To flag an issue or share your build, write to editorial@chatbotscape.com.
Methodology
RAG mechanics follow the academic and vendor documentation cited in our glossary entry; vector-search mechanics follow the sources in the vector-database entry, checked on the verification date in the frontmatter. Platform capability notes are structural (which product class puts retrieval where) rather than per-feature claims; specifics live in the individual platform reviews per our methodology. Test-set sizes and cadence numbers are operating heuristics, labeled as such, not measured benchmarks.
Last updated
13 July 2026 — Initial publication aligned to methodology v3.12.1. Next scheduled refresh: 13 October 2026.