LLM Observability· Operations
LLM Observability — Being Able to Explain a Bad Answer After It Happens
Quick answer: A large language model gives different answers to the same question, so you cannot reproduce a customer's bad experience by re-running it. Observability solves this by keeping a structured record of every request: what the bot was asked, which steps it took, what each step cost and how long it took, and what came back. Langfuse describes it as structured logs of every request that capture the exact prompt, the response, token usage, latency and any tool or retrieval steps in between. The record is also personal data in many deployments, so what you keep and for how long is a privacy decision as much as an engineering one.
The vocabulary: traces, spans, metrics, scores
| Term | What it is | Support-bot example |
|---|---|---|
| Trace | The full record of one request from start to finish | One customer message and everything the bot did to answer it |
| Span (or observation) | One step inside a trace, with start time, end time, input and output | A model call, a knowledge-base lookup, a tool call, a guardrail check |
| Metric | A number aggregated across many requests | Requests per hour, median delay, tokens per day |
| Score | A quality judgment attached to a trace | A thumbs-down from the customer, or a grade from an automatic judge |
| Session | Traces grouped into one conversation | All turns of one chat |
The OpenTelemetry project describes three signals for generative AI: traces that follow each interaction's lifecycle, metrics that aggregate things like request volume, latency and token counts, and events that capture granular moments such as individual model interactions. Langfuse's documentation lists application tracing, token and cost tracking, performance metrics, scoring, dashboards and alerts as its core pieces. Different products use different words for the same ideas, so check which one your tool means by "run," "trace" or "session."
An illustrative trace
This is invented and simplified, to show the shape of the data. It is not output from any product.
trace: "Can I get a refund on an annual plan?" total 4.1 s $0.011
span 1 guardrail check (input) 0.2 s
span 2 knowledge-base search, 4 passages retrieved 0.6 s
span 3 model call, 2,300 tokens in, 180 tokens out 3.1 s
span 4 guardrail check (output) 0.2 s
score customer thumbs-down
With this record a reviewer can open the trace for the thumbs-down, see exactly which four passages the model was given, and tell whether the problem was retrieval (wrong passages), the prompt, or the model. Without it, all you have is the final reply.
How it differs from monitoring and from chatbot analytics
Traditional monitoring asks whether the service is up and fast: error rates, server load, uptime. A bot can pass all of that and still tell customers the wrong refund policy, because the failure is in the content of the answer, not in the infrastructure. Observability adds the content and the reasoning path to the picture.
Chatbot analytics measures outcomes across conversations: containment, resolution, handoff and satisfaction, usually by business owners and on a dashboard. Observability works one level down, at individual requests, usually for whoever fixes the bot. The two share some numbers, such as cost and delay, and complement each other: analytics tells you where the problem is, a trace tells you why.
What the OpenTelemetry conventions standardize
OpenTelemetry, a vendor-neutral observability standard, has a set of generative-AI semantic conventions. Their repository says they cover generative AI operations including inference, agents, tool execution, evaluation and Model Context Protocol calls. The spans document lists recommended attributes such as input tokens, output tokens, reasoning output tokens and cache tokens, and notes that the span should cover the duration of the operation. The point of a standard is that a trace produced by one library can be read by many tools. The spans document is marked as in development, so treat attribute names as subject to change.
The privacy catch
A faithful trace contains the customer's message, and often their name, order details or health or payment context. The OpenTelemetry spans document marks capturing input and output messages as opt-in and warns the attribute is likely to contain sensitive information including user and personal data. The OpenAI Agents SDK has a setting that controls whether the inputs and outputs of model generations and function-tool outputs are stored in its traces, and the same documentation says tracing is unavailable for organizations using OpenAI's APIs under a Zero Data Retention policy. Langfuse documents masking that redacts patterns such as card numbers and email addresses before data leaves your application.
Three consequences follow. Decide what to capture before you switch tracing on. Redact before export, not after. And treat the trace store as covered by your data retention policy, subject to deletion requests such as a data subject access request. We are not lawyers; ask yours which rules apply to your customers.
Common mistakes
- Logging everything forever. More data means more cost and more personal data to protect.
- Watching averages only. A median hides the 5% of slow or failed requests customers remember.
- Treating an automatic quality score as truth. LLM-as-a-judge grades are estimates and need a human spot-check against a golden dataset.
- Tracing without a question. Decide what decision a number will change before you chart it.
FAQ
What is LLM observability?
Recording each request a language-model app handles, with its steps, prompt, response, tokens, delay and cost, so a team can explain individual answers and track quality.
Is LLM observability the same as monitoring?
No. Monitoring checks that the service is up and fast. Observability also records the content and steps of each answer, which is where most chatbot failures happen.
How is it different from chatbot analytics?
Analytics measures outcomes across conversations, such as resolution and satisfaction. Observability inspects individual requests to find out why an answer went wrong.
Do I need a special tool?
Not necessarily. Several vendors and open-source projects offer one, and OpenTelemetry provides a neutral format, but a small bot can start with the traces its platform already exposes.
Is it safe to log customer messages?
Only with a plan: capture the minimum, redact sensitive patterns, set a retention period and restrict who can read the logs.
Related terms
- Chatbot analytics — the outcome-level view that observability complements.
- Audit log — a record kept for accountability rather than debugging.
- LLM-as-a-judge — automatic quality scoring applied to traces.
- Model drift — the slow change observability helps you notice.
- Cost per conversation — a number traces make measurable.
Sources
- OpenTelemetry, Semantic conventions for generative AI (repository README) — github.com/open-telemetry/semantic-conventions-genai, read 6 October 2026. Source of the scope statement.
- OpenTelemetry, GenAI spans — github.com/open-telemetry/semantic-conventions-genai, read 6 October 2026. Source of development status, opt-in message capture, the sensitive-data warning and token attributes.
- OpenTelemetry, Generative AI observability blog post, 2024 — opentelemetry.io/blog/2024/otel-generative-ai, read 6 October 2026. Source of the three signals.
- OpenAI, Agents SDK tracing — openai.github.io/openai-agents-python/tracing, read 6 October 2026. Source of default spans, the sensitive-data setting and the Zero Data Retention limitation.
- Langfuse, LLM observability overview — langfuse.com/docs/observability/overview, read 6 October 2026. Source of the definition and feature list.
- Langfuse, Masking — langfuse.com/docs/observability/features/masking, read 6 October 2026. Source of redaction before export.
- Chatbotscape evaluation methodology. /methodology (continuously updated).