Skip to content
Chatbotscape
Editorial flat-vector illustration for How to Collect Feedback on Chatbot Answers and Use It to Improve the Bot
Published 7 min read

How to Collect Feedback on Chatbot Answers and Use It to Improve the Bot

Quick answer: Put a thumbs up and thumbs down on every answer, store each click with the exact question, answer and sources behind it, ask one short follow-up after a downvote, and read the downvotes on a fixed weekly schedule. Because few people rate, also read a random sample of unrated answers. Fix the cause (usually missing or outdated content), then re-test those questions.

Step 1: Decide what you want to learn

A rating button is cheap; deciding what to do with it is the work. Write down two or three questions the feedback should answer, for example: which topics produce the most downvotes, which answers are wrong versus merely unhelpful, and whether last month's content fixes helped. If you cannot name who will read the feedback and what they can change, wait before adding a button.

Keep it separate from the end-of-conversation score. A satisfaction rating when a chat closes, such as Intercom's five-emoji conversation rating asked at closure, measures the whole interaction; a per-answer thumb tells you which reply failed. The guide to improving CSAT covers the first, this guide the second.

Step 2: Attach every rating to its answer

A thumbs-down with no context cannot be fixed. When the customer clicks, save one record containing:

  • an answer ID and conversation ID,
  • the customer's question and the bot's reply,
  • the sources the bot used (document titles or passage IDs), if it retrieves from a knowledge base,
  • the prompt or settings version, and the time,
  • the rating and any comment.

Store it in your own database or in the platform's transcript store. Microsoft's Copilot Studio documentation, for example, says reactions are on by default for most channels, are stored in the conversation transcript table, and can be drilled into by response and comment for people with the right role; check what your platform offers. Do not store more personal data than needed, and apply the same retention limits as the transcripts; see chatbot PII handling.

Step 3: Ask one follow-up after a downvote

A thumbs-down means "wrong," "didn't help," "too vague" or "not what I asked." Right after the click, offer three or four choices plus an optional comment box:

  • The answer was wrong or out of date
  • It didn't answer my question
  • It was confusing or too long
  • I wanted a person

Keep it to one tap. Do not demand a reason before accepting the vote, and show a short thanks. "I wanted a person" is worth routing straight to a human handoff. An upvote needs no follow-up.

Step 4: Set up a weekly review

Pick a day, a reader and a time box (30 to 60 minutes). Pull that week's downvoted answers, newest first, and for each one record a cause:

  1. Missing or outdated knowledge, the answer is not in your content or is stale.
  2. Wrong passage retrieved, the right content exists but the bot fetched another one.
  3. Tone or format, correct but curt, long or badly laid out.
  4. Not the bot's fault, such as a policy the customer dislikes.

Alert on spikes only if you will act on them; a simple weekly digest of downvotes is enough for most small teams, and platform reports often provide it. Tracing tools can also show what the bot retrieved for each flagged answer, which helps with cause 2; see LLM observability and the production monitoring guide.

Step 5: Do not trust the ratings alone

Only some customers rate. A NoHold article on support bots claims about 20% of users give feedback (a vendor figure we have not verified) and says those who do skew toward strong reactions, including thumbs-downs because the underlying problem was unsolved. Your rate will differ, so measure it: ratings divided by answers.

To find the failures nobody reports, read a random sample of unrated answers each week, 20 to 40 is plenty to start, and mark each good or bad. Compare the bad share with the downvote share. If many more bad answers sit among the unrated, your ratings are understating the problem. Also look at implicit signals: the same question asked twice in a row, requests for a person, and abandoned chats.

Step 6: Fix the cause, then check

Match the fix to the cause:

  • Missing or outdated knowledge: write or update the article, then re-index. See building a chatbot knowledge base.
  • Wrong passage retrieved: adjust chunking, titles or retrieval settings, or add a clearer article for that question. See reducing hallucinations.
  • Tone or format: edit the prompt or the answer template.
  • Not the bot's fault: leave it, but make sure the bot hands off politely.

Save each fixed question, with the answer you want, in a test set and re-run it after every change; the regression testing guide and golden dataset entry explain how. A month later, compare the downvote share for those topics. Feedback does not retrain the model; the improvement comes from the changes you make.

Check the arithmetic

An invented month, not a measurement. The bot answers 2,000 questions. 160 get a rating (160 ÷ 2,000 = 8%): 120 up and 40 down, so 25% of ratings are negative.

  • The reviewer reads all 40 downvotes: 22 missing or outdated knowledge (55%), 8 wrong passage (20%), 6 tone or format (15%), 4 not the bot's fault (10%). Total 40 and 100%.
  • The team also reads a random 40 of the 1,840 unrated answers (2,000 − 160) and finds 6 bad, or 15%. If that holds, 1,840 × 0.15 = 276 bad answers were never flagged.

Two things follow: knowledge gaps are the biggest fix (22 of 40), and the visible downvotes are a small slice of the bad answers. Your numbers will differ; the method is what carries over.

Common mistakes

  • Collecting ratings nobody reads.
  • Saving the click but not the answer behind it.
  • Treating the rating rate as a satisfaction score.
  • Fixing only the downvoted question, not the topic behind it.
  • Never re-testing after a content change.
  • Ignoring unrated answers.

FAQ

How do I collect thumbs up and thumbs down feedback on chatbot answers?

Add a rating control to each answer, save the click with the question, reply, sources and settings version, and offer a one-tap reason after a downvote.

Where should the feedback be stored?

With the conversation transcript or in your own database, linked by answer ID, under the same access and retention rules as the transcript.

Are thumbs-down ratings trustworthy?

They are a useful pointer, but few customers rate and they skew toward strong reactions. Sample unrated answers as well.

How often should we review feedback?

Weekly for most teams, with a time box; daily only for a bot handling high-stakes questions.

Does feedback make the chatbot learn automatically?

Not by default. Someone reads it and changes the content, prompt or settings.

Is this the same as a CSAT survey?

No. CSAT scores a whole conversation; per-answer ratings show which reply failed. See chatbot CSAT.

Sources

About this guide

Chatbotscape launched in 2026 as an independent review site for chatbot platforms. This guide is editorial guidance built from the sources above; the steps, the review method and the worked example are our own practice with invented numbers, and we have not benchmarked any feedback tool.

Last updated

30 October 2026 — first published.