
How to Build a Live Chat QA Scorecard
Quick answer: Read 30 to 50 of your own recent chats and write down what separated the good ones from the bad. Turn that into five to seven criteria, with correctness weighted above politeness. Use simple rating scales, mark the checks that must never fail (such as identity verification) as critical, and write one line per criterion saying what a pass looks like. Review a fixed number of chats per agent each week plus the chats that went wrong, calibrate reviewers once a month, and use the scores to coach, not to rank. AI scoring in Zendesk or Intercom can widen coverage, but keep people judging whether the answer was right.
Step 1: Write criteria from your own chats
Start from chats, not from a template. Pull a mix of 30 to 50 recent chat transcripts: some with high CSAT, some with low, a few escalations and a few complaints. For each, note what made it good or bad. Patterns appear quickly: wrong policy answers, missed second questions, long silences, copy-pasted replies that did not fit.
Group those notes into five to seven criteria. Most chat teams end up with something close to this:
| Criterion | What a pass looks like | Why it matters in chat |
|---|---|---|
| Understood the issue | Answers the question actually asked, including the second one | Customers often type several questions in one message |
| Correct, complete solution | The answer matches policy and the product, with nothing missing | The criterion CSAT cannot check |
| Policy and security | Identity verified before account details; no personal data pasted carelessly | One failure can be a data incident |
| Tone and empathy | Reflects the customer's situation in their own terms | Text loses tone; generic phrases read as scripted |
| Clear writing | Short messages, plain words, correct spelling, useful links | The whole conversation is written |
| Chat handling | No long silent gaps, warned before a hold, clean transfer or human handoff | Agents juggle several chats at once |
Call Centre Helper's Neil Martin suggests checking your criteria with one question: could a template reply pass? If a pasted canned response that ignores the customer's details would score full marks, the criteria are too loose.
Step 2: Choose rating scales and weights
Keep scales simple. Pass/fail is fastest and the most consistent between reviewers; a 3-point scale (meets, partly, misses) adds room for "right answer, missing a step". Zendesk QA offers binary, 3-point, 4-point, 5-point and custom scales; Intercom lets each criterion have its own rating options with a score such as 100, 50 or 0 percent, or "Not scored". Allow N/A for criteria that do not always apply, such as verification on a pre-sales chat.
Then weight the criteria. Both Zendesk and Intercom use relative weights from 0 to 100. A worked example, with a made-up review:
| Criterion | Weight | Rating | Points |
|---|---|---|---|
| Understood the issue | 20 | 100% | 20 |
| Correct, complete solution | 30 | 50% | 15 |
| Policy and security (critical) | 15 | 100% | 15 |
| Tone and empathy | 15 | 100% | 15 |
| Clear writing | 10 | 50% | 5 |
| Chat handling | 10 | 0% | 0 |
| Total | 100 | 70% |
If policy and security were N/A on this chat, it would usually drop out of both sides: 55 points out of 85 is 64.7 percent. Confirm how your tool treats N/A before you publish targets.
Step 3: Mark critical criteria and add reasons
A critical criterion fails the review on its own. Keep this list short, for things that must never happen: sharing account details without verification, promising a refund outside policy, a legal or compliance statement. In Zendesk QA a critical category rated below 50 percent fails either the whole review (score 0 percent) or just its group; in Intercom a failing rating on a critical criterion fails the review.
Add reasons (Zendesk calls them root causes, Intercom rating reasons) to the criteria that fail most: "wrong help article", "policy unclear", "tool did not show order status". Reasons turn a score into something you can fix, and they often point at the knowledge base or tooling rather than the agent. Tag patterns can be cross-checked with your conversation tags.
Step 4: Decide how many chats to review, and which
There is no standard quota. Practitioners quoted by Call Centre Helper range from four contacts per agent a month to 50 or more; the most common answer in its poll was more than 10 per agent a month. Pick a number you can sustain, and split it into two streams:
- Random sample per agent. The same number each week, so scores are comparable over time. New agents get more until their scores settle.
- Targeted reviews. Chats with low CSAT, escalations, complaints, very long handle times or reopened issues. Keep these out of the agent's average, or label them, because they are chosen for being difficult.
An invented example of the arithmetic. A team of 6 agents handles about 3,000 chats a month and reviews 8 random chats per agent a month: 6 × 8 = 48 reviews, which is 1.6 percent of chats. At an assumed 10 minutes per review, that is 480 minutes, or 8 hours of reviewer time a month, before targeted reviews.
Be honest about what that sample can show. With 48 reviews, a team-wide pass rate near 50 percent is only accurate to roughly ±14 points at 95 percent confidence (1.96 × √(0.25 ÷ 48) ≈ 0.14), and 8 chats per agent says even less about one person. The same formula gives about ±5 points for 400 reviews, which matches the sampling note in Call Centre Helper. Use per-agent scores as a prompt for a coaching conversation, not as a verdict.
Step 5: Calibrate reviewers and set it up in your tool
Before the first real round, have everyone who reviews score the same five chats separately, then compare. Where scores differ, agree what the criterion means and rewrite its one-line description. Repeat monthly with new chats. Let agents see their reviews and raise a dispute; Intercom has a dispute workflow in closed beta, and a simple shared form works too.
Then build it in your help desk. As documented on 18 November 2026:
| Tool | Where | What to know |
|---|---|---|
| Zendesk QA (add-on) | Quality assurance, Settings, Account, Scorecards | Weights, critical categories and root causes apply to manual reviews only; AutoQA categories (greeting, empathy, spelling and grammar, closing, solution offered, tone, readability, comprehension) must be assigned to a scorecard and score solved or closed tickets |
| Intercom (Pro add-on, human teammate QA) | Monitors and scorecards | A Monitor selects closed conversations by random sample, filters or plain-language flag criteria, and can cap reviews (for example 10 a day); each criterion's description is the AI's scoring prompt; teammates can override AI scores |
If your help desk has no QA feature, a spreadsheet with one row per review and a column per criterion works for a small team. Compare tools in our Intercom review and see Zendesk.
Step 6: Use AI scoring carefully and turn scores into coaching
AI scoring, a form of LLM-as-a-judge, is good at clear checks across every chat: greeting, closing, spelling, whether a solution was offered. Use it to cover the whole volume and to pick chats for people to review. It is weaker at whether the solution was correct for your product and policy. Intercom routes a conversation to manual review unless every criterion is AI-scored with confidence, and says its scorecards are not meant for employment decisions based only on AI scores without human review.
Close the loop every week:
- One-to-one coaching on one or two criteria per agent, using the actual chats.
- Team fixes where the same reason repeats: update the help article, the macro or the bot answer. Recurring wrong answers often mean the knowledge base is out of date.
- Compare with CSAT each month. Call Centre Helper reports that only 33 percent of 194 professionals surveyed said quality scores matched CSAT well; a large gap means the scorecard is measuring the wrong things. Our guide to improving chatbot CSAT covers the survey side.
- Review the bot too. If a bot answers first, score a sample of its conversations with the same correctness criterion, alongside its resolution rate.
Common mistakes
- Copying a call-centre form with phone items (hold music, call opening) and no chat items (reply gaps, transfers, links).
- Weighting politeness above correctness, so a friendly wrong answer scores well.
- Too many critical criteria, so most reviews fail and scores stop meaning anything.
- Mixing targeted reviews into averages, which makes agents who get hard chats look worse.
- Skipping calibration, so the score depends on the reviewer.
- Trusting AI scores alone for correctness or for decisions about people.
FAQ
What should a live chat QA scorecard include? Five to seven weighted criteria: understanding the issue, a correct and complete solution, policy and security, tone, clear writing and chat handling, each with a one-line description of a pass.
How should QA criteria be weighted? Put most weight on correctness and understanding, less on tone and writing, and make the never-fail checks critical rather than heavily weighted. Zendesk and Intercom both use relative weights from 0 to 100.
How many chats should I review per agent each week? The sources we read do not agree on a number. Pick a fixed random sample you can sustain, review more for new agents, and add targeted reviews of escalations and low-rated chats.
What is QA calibration? Reviewers score the same chats separately, compare, and agree what each criterion means, so an agent's score does not depend on who reviewed them. Do it before launch and then monthly.
Can AI score support chats automatically? Yes. Zendesk AutoQA and Intercom Monitors can score conversations with AI, but both keep human review and overrides. Use AI for coverage and clear checks, people for correctness and coaching.
How is this different from testing a chatbot? QA reviews real conversations after they happen. Testing a bot before launch uses prepared test cases; see the chatbot QA testing protocol.
Sources
- Zendesk, Setting up a scorecard — support.zendesk.com/hc/en-us/articles/7043760215194, read 18 November 2026. Source of the settings path, rating scales, weights from 0 to 100, N/A, critical categories failing the review or group below 50 percent, root causes, and settings applying to manual reviews only.
- Zendesk, Understanding autoscoring categories — support.zendesk.com/hc/en-us/articles/7043747123354, read 18 November 2026. Source of the AutoQA categories and scoring of solved or closed tickets.
- Intercom, Getting started with human teammate QA in Monitors — intercom.com/help/en/articles/14840301, read 18 November 2026. Source of Monitors, sampling and review caps, rating options, weights, critical criteria, AI scoring prompts and overrides, the dispute beta, the Pro add-on and the note on employment decisions.
- Call Centre Helper, How to Assess Quality on Email and Live Chat in the Contact Centre (Neil Martin) — callcentrehelper.com/assess-quality-email-live-chat-127311.htm, read 18 November 2026. Source of the template-reply test, calibration advice and the 33 percent QA-CSAT match figure.
- Call Centre Helper, Determining the Right Number of Calls Analysed (Jonty Pearce) — callcentrehelper.com/number-of-calls-analysed-207022.htm, read 18 November 2026. Source of the range of review volumes, the poll answer and the 400-contact sampling note.
- Chatbotscape evaluation methodology. /methodology (continuously updated).