Skip to content
Chatbotscape
Zendesk's scorecard setup and autoscoring categories articles, Intercom's article on human teammate QA in Monitors, and Call Centre Helper's articles on assessing chat quality and on how many contacts to analyse were read on 18 November 2026.
Chat quality assurance· Support operations
Chat quality assurance (chat QA) is the practice of reviewing a sample of live chat or messaging conversations and scoring each one against a written scorecard, so a support team can see whether agents gave correct, complete and well-handled answers, and coach them on what to change.
By Chatbotscape Editorial· Methodology· Published 19 November 2026· Updated 19 November 2026

Chat Quality Assurance — Scoring Support Chats Against a Standard

Quick answer: Chat QA means someone (or a tool) reads finished support chats and rates them against the same list of criteria every time. The list is the scorecard: did the agent understand the problem, give the right answer, follow policy, write clearly and handle the chat well. Each criterion has a rating scale and a weight, and some can be marked critical so one failure fails the whole review. Teams use the scores to coach agents and to find problems that customer surveys miss. Help desks such as Zendesk and Intercom now add AI scoring, which can rate every chat but still needs people checking it.

What a chat QA scorecard contains

A scorecard is a short form a reviewer fills in for each chat. The parts are the same in most tools:

PartWhat it isExample
Criteria (categories)The things you checkUnderstood the issue; correct solution; tone; spelling and grammar; closing
Rating scaleHow each criterion is ratedPass/fail; 3-point; 5-point; N/A when it does not apply
WeightHow much a criterion counts towards the totalCorrect solution counts three times as much as the greeting
Critical criterionA check that fails the review on its ownIdentity not verified before sharing account details
Reasons (root causes)Why the rating was givenWrong article sent; policy not known; tool missing

Zendesk QA, for example, offers binary, 3-point, 4-point, 5-point and custom scales, weights from 0 to 100 (default 10), an N/A option, and critical categories that fail either the whole review or one group of criteria when rated below 50 percent. Intercom's scorecards use the same ideas: weights from 0 to 100, an optional pass threshold, and critical criteria that fail the review.

What chat QA checks that other metrics do not

CSAT tells you how the customer felt; QA tells you whether the agent did the job right. They often disagree. Call Centre Helper reports a survey of 194 contact centre professionals in which only 33 percent said their quality scores were a good match with CSAT. A customer can rate a friendly chat highly even when the agent gave a wrong refund date, and a correct answer can still annoy someone who waited too long.

Chat also has its own things to check that phone QA forms miss: reply gaps while the agent juggles other chats (see chat concurrency), whether a canned response was edited to fit the question, links and attachments, and how cleanly a chat moved from a bot to a person in the human handoff. Neil Martin, writing in Call Centre Helper, suggests a simple test for written channels: a reply that reads like a template should not be able to pass.

How a QA score is calculated

The usual method is a weighted average of the criteria that apply. This is our own worked example, not a vendor formula:

CriterionWeightRatingPoints
Understood the issue20100%20
Correct, complete solution3050%15
Policy and verification (critical)15100%15
Tone and empathy15100%15
Clear writing1050%5
Chat handling (reply gaps, transfers)100%0
Total10070%

If the agent had skipped identity verification, the critical criterion would fail and the review would score 0 percent, whatever the other ratings. If a criterion is marked N/A, it is usually left out of both the points and the weights. Check how your own tool does it, because the scorecard's weights and critical settings may apply to manual reviews only: Zendesk says they do not change its automatic quality score.

How many chats to review

There is no standard. Call Centre Helper collected answers from practitioners ranging from four contacts per agent a month to 50 or more, and its most common poll answer was more than 10 per agent a month. One contributor notes that about 400 randomly sampled contacts can estimate a whole team's quality to within roughly ±5 percent, but that this sampling logic does not tell you how many to review per agent. In practice, teams pick a fixed number per agent, add more for new agents or those with low scores, and add targeted reviews of chats with low CSAT, escalations or complaints.

Calibration

Two reviewers scoring the same chat should land close together. Calibration means reviewers score the same few chats separately, compare, and agree what each criterion means, then write that down in the scorecard's descriptions. Without it, an agent's score depends on who reviewed them, and coaching loses trust.

Where AI scoring fits

AI QA tools read chat transcripts and rate them automatically, usually with a language model applying the scorecard, a version of LLM-as-a-judge. As documented on 18 November 2026:

ToolFeatureWhat the documentation says
ZendeskAutoQA in Zendesk QA (add-on)Default categories include greeting, empathy, spelling and grammar, closing, solution offered, tone, readability and comprehension; scores solved or closed tickets, "100% of your ticket volume" when categories are active on a scorecard
IntercomMonitors and scorecards (human teammate QA, Pro add-on)A Monitor picks conversations (random sample, filters or plain-language flag criteria); each criterion's description is the AI's scoring prompt; AI and human scoring can be mixed; teammates can override AI scores

Both vendors keep people in the loop. Intercom sends a conversation to manual review unless every criterion is AI-scored with confidence, and says its scorecards are not meant for employment decisions based only on AI scores without human review. See our Intercom review and Zendesk. AI scoring is strongest on checks with a clear answer (greeting, closing, grammar) and weakest on whether the solution was actually right, which still needs someone who knows the product.

FAQ

What is chat QA in customer service? Reviewing a sample of finished chats and scoring each one against a fixed scorecard, so the team can coach agents and spot recurring problems.

What should a chat QA scorecard include? A few weighted criteria: understanding the issue, a correct and complete solution, policy and security steps, tone, clear writing and chat handling. Mark the ones that must never fail as critical.

How is QA different from CSAT? CSAT is the customer's rating; QA is your own check of whether the agent was right and followed process. Track both, because they often disagree.

How many chats should be reviewed per agent? The sources we read do not agree on a number. A fixed weekly sample per agent, more for new agents, plus targeted reviews of escalations and low-rated chats is a common pattern.

Can AI replace human QA reviewers? It can score more chats and handle clear-cut checks, but both Zendesk and Intercom keep human review and overrides. Use AI to choose and pre-score chats, and people to judge correctness and coach.

Does chat QA apply to chatbot conversations? Yes. Zendesk scores bots separately from agents, and the same scorecard ideas apply to bot answers, alongside the resolution rate.

Sources

  • Zendesk, Setting up a scorecard — support.zendesk.com/hc/en-us/articles/7043760215194, read 18 November 2026. Source of the rating scales, weights from 0 to 100 (default 10), the N/A option, critical categories below 50 percent failing the review or group, root causes, and settings applying to manual reviews only.
  • Zendesk, Understanding autoscoring categories — support.zendesk.com/hc/en-us/articles/7043747123354, read 18 November 2026. Source of the AutoQA default categories, scoring after a ticket is solved or closed, the 100 percent coverage condition and bots scored separately.
  • Intercom, Getting started with human teammate QA in Monitors — intercom.com/help/en/articles/14840301, read 18 November 2026. Source of Monitors and scorecards, sampling options, AI and human scoring, overrides, weights, pass threshold, critical criteria, the Pro add-on and the note on employment decisions.
  • Call Centre Helper, How to Assess Quality on Email and Live Chat in the Contact Centre (Neil Martin) — callcentrehelper.com/assess-quality-email-live-chat-127311.htm, read 18 November 2026. Source of the 33 percent QA-CSAT match figure and the template-reply test.
  • Call Centre Helper, Determining the Right Number of Calls Analysed (Jonty Pearce) — callcentrehelper.com/number-of-calls-analysed-207022.htm, read 18 November 2026. Source of the range of review volumes, the most common poll answer and the 400-contact sampling note.
  • Chatbotscape evaluation methodology. /methodology (continuously updated).