Chat quality assurance· Support operations
Chat Quality Assurance — Scoring Support Chats Against a Standard
Quick answer: Chat QA means someone (or a tool) reads finished support chats and rates them against the same list of criteria every time. The list is the scorecard: did the agent understand the problem, give the right answer, follow policy, write clearly and handle the chat well. Each criterion has a rating scale and a weight, and some can be marked critical so one failure fails the whole review. Teams use the scores to coach agents and to find problems that customer surveys miss. Help desks such as Zendesk and Intercom now add AI scoring, which can rate every chat but still needs people checking it.
What a chat QA scorecard contains
A scorecard is a short form a reviewer fills in for each chat. The parts are the same in most tools:
| Part | What it is | Example |
|---|---|---|
| Criteria (categories) | The things you check | Understood the issue; correct solution; tone; spelling and grammar; closing |
| Rating scale | How each criterion is rated | Pass/fail; 3-point; 5-point; N/A when it does not apply |
| Weight | How much a criterion counts towards the total | Correct solution counts three times as much as the greeting |
| Critical criterion | A check that fails the review on its own | Identity not verified before sharing account details |
| Reasons (root causes) | Why the rating was given | Wrong article sent; policy not known; tool missing |
Zendesk QA, for example, offers binary, 3-point, 4-point, 5-point and custom scales, weights from 0 to 100 (default 10), an N/A option, and critical categories that fail either the whole review or one group of criteria when rated below 50 percent. Intercom's scorecards use the same ideas: weights from 0 to 100, an optional pass threshold, and critical criteria that fail the review.
What chat QA checks that other metrics do not
CSAT tells you how the customer felt; QA tells you whether the agent did the job right. They often disagree. Call Centre Helper reports a survey of 194 contact centre professionals in which only 33 percent said their quality scores were a good match with CSAT. A customer can rate a friendly chat highly even when the agent gave a wrong refund date, and a correct answer can still annoy someone who waited too long.
Chat also has its own things to check that phone QA forms miss: reply gaps while the agent juggles other chats (see chat concurrency), whether a canned response was edited to fit the question, links and attachments, and how cleanly a chat moved from a bot to a person in the human handoff. Neil Martin, writing in Call Centre Helper, suggests a simple test for written channels: a reply that reads like a template should not be able to pass.
How a QA score is calculated
The usual method is a weighted average of the criteria that apply. This is our own worked example, not a vendor formula:
| Criterion | Weight | Rating | Points |
|---|---|---|---|
| Understood the issue | 20 | 100% | 20 |
| Correct, complete solution | 30 | 50% | 15 |
| Policy and verification (critical) | 15 | 100% | 15 |
| Tone and empathy | 15 | 100% | 15 |
| Clear writing | 10 | 50% | 5 |
| Chat handling (reply gaps, transfers) | 10 | 0% | 0 |
| Total | 100 | 70% |
If the agent had skipped identity verification, the critical criterion would fail and the review would score 0 percent, whatever the other ratings. If a criterion is marked N/A, it is usually left out of both the points and the weights. Check how your own tool does it, because the scorecard's weights and critical settings may apply to manual reviews only: Zendesk says they do not change its automatic quality score.
How many chats to review
There is no standard. Call Centre Helper collected answers from practitioners ranging from four contacts per agent a month to 50 or more, and its most common poll answer was more than 10 per agent a month. One contributor notes that about 400 randomly sampled contacts can estimate a whole team's quality to within roughly ±5 percent, but that this sampling logic does not tell you how many to review per agent. In practice, teams pick a fixed number per agent, add more for new agents or those with low scores, and add targeted reviews of chats with low CSAT, escalations or complaints.
Calibration
Two reviewers scoring the same chat should land close together. Calibration means reviewers score the same few chats separately, compare, and agree what each criterion means, then write that down in the scorecard's descriptions. Without it, an agent's score depends on who reviewed them, and coaching loses trust.
Where AI scoring fits
AI QA tools read chat transcripts and rate them automatically, usually with a language model applying the scorecard, a version of LLM-as-a-judge. As documented on 18 November 2026:
| Tool | Feature | What the documentation says |
|---|---|---|
| Zendesk | AutoQA in Zendesk QA (add-on) | Default categories include greeting, empathy, spelling and grammar, closing, solution offered, tone, readability and comprehension; scores solved or closed tickets, "100% of your ticket volume" when categories are active on a scorecard |
| Intercom | Monitors and scorecards (human teammate QA, Pro add-on) | A Monitor picks conversations (random sample, filters or plain-language flag criteria); each criterion's description is the AI's scoring prompt; AI and human scoring can be mixed; teammates can override AI scores |
Both vendors keep people in the loop. Intercom sends a conversation to manual review unless every criterion is AI-scored with confidence, and says its scorecards are not meant for employment decisions based only on AI scores without human review. See our Intercom review and Zendesk. AI scoring is strongest on checks with a clear answer (greeting, closing, grammar) and weakest on whether the solution was actually right, which still needs someone who knows the product.
FAQ
What is chat QA in customer service? Reviewing a sample of finished chats and scoring each one against a fixed scorecard, so the team can coach agents and spot recurring problems.
What should a chat QA scorecard include? A few weighted criteria: understanding the issue, a correct and complete solution, policy and security steps, tone, clear writing and chat handling. Mark the ones that must never fail as critical.
How is QA different from CSAT? CSAT is the customer's rating; QA is your own check of whether the agent was right and followed process. Track both, because they often disagree.
How many chats should be reviewed per agent? The sources we read do not agree on a number. A fixed weekly sample per agent, more for new agents, plus targeted reviews of escalations and low-rated chats is a common pattern.
Can AI replace human QA reviewers? It can score more chats and handle clear-cut checks, but both Zendesk and Intercom keep human review and overrides. Use AI to choose and pre-score chats, and people to judge correctness and coach.
Does chat QA apply to chatbot conversations? Yes. Zendesk scores bots separately from agents, and the same scorecard ideas apply to bot answers, alongside the resolution rate.
Related terms
- Chatbot CSAT — the customer's rating that QA complements.
- Chat transcript — the record a reviewer reads.
- Average handle time — speed, which QA balances against correctness.
- First contact resolution — whether the issue was solved in one go, often a QA criterion.
- LLM-as-a-judge — the technique behind most AI QA scoring.
- Conversation tagging — tags that help select chats for review.
- Live chat — the channel being reviewed.
Sources
- Zendesk, Setting up a scorecard — support.zendesk.com/hc/en-us/articles/7043760215194, read 18 November 2026. Source of the rating scales, weights from 0 to 100 (default 10), the N/A option, critical categories below 50 percent failing the review or group, root causes, and settings applying to manual reviews only.
- Zendesk, Understanding autoscoring categories — support.zendesk.com/hc/en-us/articles/7043747123354, read 18 November 2026. Source of the AutoQA default categories, scoring after a ticket is solved or closed, the 100 percent coverage condition and bots scored separately.
- Intercom, Getting started with human teammate QA in Monitors — intercom.com/help/en/articles/14840301, read 18 November 2026. Source of Monitors and scorecards, sampling options, AI and human scoring, overrides, weights, pass threshold, critical criteria, the Pro add-on and the note on employment decisions.
- Call Centre Helper, How to Assess Quality on Email and Live Chat in the Contact Centre (Neil Martin) — callcentrehelper.com/assess-quality-email-live-chat-127311.htm, read 18 November 2026. Source of the 33 percent QA-CSAT match figure and the template-reply test.
- Call Centre Helper, Determining the Right Number of Calls Analysed (Jonty Pearce) — callcentrehelper.com/number-of-calls-analysed-207022.htm, read 18 November 2026. Source of the range of review volumes, the most common poll answer and the 400-contact sampling note.
- Chatbotscape evaluation methodology. /methodology (continuously updated).