Skip to content
Chatbotscape
Editorial flat-vector illustration for How to Get Valid JSON from a Chatbot's Language Model: Schema, Checks, Retries
Published 8 min read

How to Get Valid JSON from a Chatbot's Language Model

Schema, Checks, Retries

Quick answer: Write down the exact fields you need as a schema, use your provider's structured-output setting so the model's reply is held to that schema, then still treat the reply as untrusted: check why generation ended, detect refusals, validate the values against your own data, retry once on failure, and hand the conversation to a person if the retry also fails. In our invented example below, a chatbot told only to "reply in JSON" produced 8 unusable replies in 50 test messages; a strict schema removed all 8, and a value check caught the 5 replies that were well formed but wrong.

Step 1: Decide what the software needs, and nothing more

Start from the action, not the model. If the next step is "look up an order and tell the customer its status", the software needs an order number and perhaps an email. List each field with its type and say whether it can be missing. Keep the list short: every extra field is another chance for an error and another thing to test. Also decide what the model should do when a value is not in the message. Under strict rules, where every field is listed as required, a common way to handle a missing value is to let that field be null and treat null as "ask the customer".

Step 2: Write the schema and switch on the strict setting

Express the list as a JSON Schema. For a closed set of choices, such as a category, use an enumeration of allowed labels instead of free text, so a reply cannot invent a sixth category. Then use the structured-output setting instead of only describing JSON in the prompt.

OpenAI's documentation separates three levels: JSON mode guarantees valid JSON but not your fields; structured outputs with strict set to true are designed so the reply matches your schema; and a plain instruction guarantees nothing. It recommends structured outputs over JSON mode where available. Its examples set additionalProperties to false and list every property as required. Only a subset of JSON Schema is supported, and OpenAI notes that schemas supplied this way are not eligible for zero data retention, so tell whoever owns privacy before you ship.

Step 3: Handle the three ways a reply can still not be data

A strict schema removes most malformed replies. Three cases still reach your code, and each needs a branch.

  1. A refusal. OpenAI reports a refusal in a separate field of the response. Check it first; if present, do not parse, and send the customer the normal fallback message.
  2. A cut-off reply. If generation stops early, for example at a length limit, the reply may not match the schema. Check the reason generation ended. If it was not a normal finish, treat the reply as failed. A larger output limit is the usual first fix.
  3. A slow first call. The first request with a new schema takes longer while it is processed: OpenAI says typically under ten seconds and up to a minute for complex schemas, cached afterwards. Send one warm-up request after each deploy, so no customer waits for it.

Step 4: Validate the values, not just the shape

This step is the one most guides skip. A schema guarantees structure; it does not know whether the order number was copied correctly. Our rule: every value that triggers an action gets checked against something you hold. An order number must exist in your system. A date must be in the future. An email must belong to the account in the session. If a check fails, do not act on it; ask the customer to confirm.

Libraries in several programming languages can define the schema in code and parse the reply into typed objects; OpenAI's SDKs offer helpers of this kind for Python and JavaScript. We have not compared libraries, so choose one your developer already knows.

Step 5: Retry once, with the error, then stop

If validation fails, one retry is usually enough. Send the model the same message plus the specific problem, for example "the order number you returned was not found; return the number exactly as the customer wrote it, or null". Our rule is one retry. A second failure on the same message means the message is unclear, and more automated tries rarely help while adding delay and cost.

Step 6: Fall back to a person or a question

When the retry fails, never guess. Either ask the customer a plain question ("Could you send me the order number from your confirmation email?") or pass the conversation to an agent with the original message attached. Log every fallback with its reason (refusal, cut-off, failed check), because the pattern tells you which step to improve. A fallback rate that keeps climbing usually means the schema asks for something customers do not provide. See monitor an LLM chatbot in production for what to log.

Worked example (all numbers invented)

A small shop's chatbot must extract an order number and a request type (status, return or cancel) from customer messages. The team builds a test set of 50 real-style messages and runs two setups. This is an illustration with invented counts, not a measurement of any product.

SetupReplies that parse and match the schemaReplies that do notCorrect values among those that match
Prompt says "reply in JSON"42840
Strict schema, same prompt50045

Of the 8 failures in the first row, 5 had a sentence of chat text wrapped around the JSON, 2 left out the request type, and 1 used a label ("refund") that was not one of the three allowed. That is 5 + 2 + 1 = 8, and 42 + 8 = 50. With the strict schema, all 50 match, because the labels are limited to the three allowed values and no extra text can appear.

The second row also shows the limit. Only 45 of the 50 have correct values, so 5 replies are well formed and wrong: in each, the order number was mistyped by the model when the customer wrote it with spaces or a leading hash sign. Step 4 catches these, because those numbers do not exist in the shop's system, and the bot asks the customer to confirm instead of reporting the wrong order. The team then adds one line to the prompt about copying the number exactly, reruns the 50 messages and checks that none of the 45 correct ones broke. This is the same discipline described in version and release chatbot prompts.

Common mistakes

  • Trusting "reply in JSON" in the prompt. It works until it does not, and the failures break automation silently.
  • Skipping the refusal and cut-off checks. The reply then reaches the parser as text and throws an error in front of a customer.
  • Treating a valid shape as a correct answer. Validate values against your own data before acting.
  • Making every field required with no way to say "unknown". The model fills the gap with a guess. Allow null.
  • Retrying without limit. One retry, then a person.
  • Not testing after a schema change. Rerun the fixed test set, as with any prompt change.

FAQ

How do I get an LLM to return valid JSON?

Define the fields as a JSON Schema and use your provider's structured-output setting rather than only asking for JSON in the prompt. Then check for refusals and cut-off replies and validate the values.

Is JSON mode enough?

JSON mode makes the reply valid JSON but does not guarantee your fields. If your code needs specific fields, use a schema-based setting where your provider offers one.

What do I do when the JSON is malformed anyway?

Check why generation ended, retry once with the specific error, and if that fails ask the customer a question or pass the chat to a person.

Does a schema stop wrong answers?

No. It fixes the shape. Check values such as order numbers against your own system before acting on them.

Should the model call a function or return structured output?

Use function calling when the model needs your system to do something, and a structured reply when you need data back. Many bots use both.

How many test messages do I need?

There is no fixed number. Start with a set that covers your real message types, including messy ones, and grow it from failures. A golden dataset is the usual name for such a set.

Sources

About this guide

Chatbotscape launched in 2026 as an independent review site for chatbot platforms. This guide is editorial guidance built from provider documentation; the six-step order, the one-retry rule and the worked example are our own practice with invented numbers, and we have not benchmarked any library or provider.

Last updated

17 October 2026 — first published.