How to Summarize Chatbot Conversations for a Human Handoff

Quick answer: Decide what an agent needs in the first ten seconds, fix that as a short template, and have the model fill it in from the transcript only, leaving a field blank when the chat does not say. Put the note where the agent already looks, keep the full transcript one click away, and each week compare a sample of notes with their transcripts. Count missing and wrong facts separately, because a wrong fact does more harm than a gap.

Step 1: Ask agents what they need first

Sit with two or three agents and look at five recent handoffs. For each, note what they scrolled for. It is usually the same few things: what the customer wants, what the bot already tried, an order or account number and what is blocking. Write those down. Everything else in a note is clutter.

If you use a platform, check how it builds its note. Intercom's community post describes AI Summaries posted as an internal note at handoff, switched on by an "Add summary note" action in a triage workflow. Ada's documentation for Zendesk Chat says it can attach a summary built from up to three of the customer's text messages, and attaches none if it cannot tell the intent. Different inputs produce different notes, so read what yours uses.

Step 2: Fix a template

A fixed shape is easier to read and easier to check than free prose. A template we use as a starting point, which is our own practice and not a standard:

• Issue: one sentence, in the customer's terms.
• Already done: what the bot did or told them.
• Open: what is left and why the bot stopped (for example, "customer asked for a person" or "refund above limit").
• Facts: order number, dates, amounts, product, exactly as given.
• Customer's last message: copied verbatim.

The verbatim last message is a cheap safeguard: it is extractive, so it cannot be invented, and it lets the agent spot a summary that has drifted.

Step 3: Prompt the model from the transcript only

Give the model the transcript and the template, and tell it plainly what to do when information is missing. Wording along these lines, to adapt and test on your own data:

Summarize the conversation below for a support agent using the template. Use only what is written in the conversation. If a field is not stated, write "not stated". Do not guess order numbers, amounts or promises. Quote the customer's last message exactly.

Three practical points.

1. Ask for structure. If your tool supports it, request a fixed set of fields so the note can be checked by code. The JSON output guide covers this.
2. Mind the length. A very long chat may need older turns shortened first. The context window guide explains the limits.
3. Pass facts from your systems, not the model. An order number that your bot already stored should be copied from that record, not retyped by the model.

Step 4: Put the note where the agent looks

A note nobody sees saves nothing. Show it at the top of the conversation in the agent's inbox or ticket, labeled as AI-written, with the full transcript one click away. Intercom's post describes an internal note, and Ada's description places the summary after the last transcript message, so placement differs between products. Whatever you choose, test it with a real agent. Combine it with a warm transfer, so the customer is told that the agent has the details. Wording for that message is in the escalation playbook.

If you also give agents live suggestions, see the agent assist guide; a summary is one part of that kind of tool.

Step 5: Check accuracy every week

Pull 20 to 50 handoffs at random. Compare each note with its transcript and mark it one of three ways:

1. Correct and complete.
2. Missing something the agent needed.
3. Contains a wrong fact or an invented action.

Track the wrong-fact share most closely. A reading by a person is the most reliable check. You can add an LLM judge to cover more notes, but test it against your own human marks first. Do not rely on an overlap score such as ROUGE alone: a 2019 study of news summaries found it correlated only weakly with human ratings of factual consistency, and in that study 60 of 200 summaries from abstractive models had consistency problems. That was news, not support chat, so measure your own. The hallucination guide lists ways to cut invented content, and the QA testing protocol shows how to keep the test set.

Also ask agents directly. A two-question prompt in the inbox, "Was this note useful?" and "Anything wrong?", gives you examples to fix.

Step 6: Handle personal data and records

A note can copy names, addresses and payment fragments into a ticket. Tell the model not to include card numbers or government IDs, and add a code check that removes them before saving. Decide how long notes are kept and who can read them, using the chatbot security and PII guide. Keep the transcript and the note together so a dispute can be traced. This is general guidance, not legal advice.

Check the arithmetic

An invented example, not a measurement. A team samples 50 notes in a week.

• 41 are correct and complete, 6 miss a needed fact and 3 contain a wrong fact. Check: 41 + 6 + 3 = 50. Shares: 82%, 12% and 6%.
• The team handles 400 handoffs a month. If 6% held across all of them, 400 × 0.06 = 24 notes with a wrong fact.
• Suppose a note saves an agent about 75 seconds of reading. 400 × 75 = 30,000 seconds, which is 500 minutes or about 8.3 hours a month. That 75-second figure is assumed.

On these made-up numbers the notes save time, and the 24 faulty ones show where the effort should go: tighten the prompt for wrong facts, and keep the verbatim last message so agents can catch them.

Common mistakes

• No template. Free prose is long and hard to check.
• Letting the model guess. Missing fields must read "not stated".
• Hiding the transcript. Agents need a way to verify the note.
• Checking only with overlap scores. They can miss factual errors.
• Skipping the agent view test. A note in the wrong place is never read.
• Copying sensitive data into tickets. Filter before saving.

FAQ

How do I make a chatbot summarize a conversation for a human agent?

Fix a short template, have the model fill it from the transcript only, show it at the top of the agent's view and keep the transcript linked.

What should a handoff summary include?

The issue, what the bot already did, what is still open and the key facts such as an order number, plus the customer's last message verbatim.

How do I stop the model inventing details in a summary?

Tell it to use only the conversation and write "not stated" for gaps, copy known facts from your own records, and check a weekly sample.

How do I measure summary quality?

Compare a random sample with transcripts, mark each correct, missing or wrong, and track the wrong-fact share. Add an LLM judge only after testing it against human marks.

Do customers still need to repeat themselves?

Less often, if the agent reads the note and the customer is told the agent has the details. A note that is wrong or unseen does not help.