LLM observability: seeing what your AI application really does
LLM observability covers the traces, metrics and logs that show, request by request, what an application built on a language model is doing: which prompt went out, which response came back, which tools were called, what it cost and how long it took.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|
Definition
Put simply: it is a flight recorder for an AI application. An LLM is non-deterministic, and an AI agent may chain many steps before answering. Without detailed records, there is no way to tell why it went wrong or where the budget went.
An observability setup captures, for every request, a trace made of steps (spans): the system prompt and the question, the retrieved passages, every tool call with its arguments, the final answer, the latency of each step and the tokens consumed. Specialized tools, some of them open source such as Langfuse, display these traces as a tree.
Toward a shared vocabulary
The OpenTelemetry project, the open standard for software telemetry, maintains semantic conventions for generative AI: spans, metrics and events for model clients, agents and the MCP protocol. They include attributes such as gen_ai.request.model (the requested model) and gen_ai.usage.input_tokens (input tokens). The benefit: you can switch analysis tools without re-instrumenting your code. As of October 22, 2026, these conventions are still at the “development” stage.
A concrete example
A travel booking agent’s costs climb over a single week. The traces reveal that after an update, the agent keeps calling the flight search tool in a loop whenever a date is ambiguous, resending the full history to the model each time. The fix: ask the user a clarifying question and cap the number of iterations. Subsequent traces confirm things are back to normal. To anticipate this kind of drift, our agent cost calculator helps estimate the budget based on price per token.
Why it matters
- Debugging: pinpoint the exact step where the agent went off track.
- Cost control: spot oversized prompts, loops and pointless calls.
- Continuous evaluation: plug automated checks into the traces, such as an LLM-as-a-judge that scores a sample of answers.
Pitfalls and misconceptions
- “Logs are enough.” Isolated log lines don’t show the causal chain of an agent’s steps.
- “Record everything and sort it out later.” Traces hold user data: think privacy rules such as the GDPR, redaction and retention limits.
- “Observing is evaluating.” Traces show what happened; judging whether it was right takes criteria and tests.
Related terms: agent evaluation, orchestration, guardrails.
Related terms
- Agent evaluation · Methods for measuring whether an AI agent really completes a task end to end, reliably, safely and at an acceptable cost.
- LLM-as-a-judge · An evaluation method in which a language model scores or compares another model's answers against a rubric, standing in for a human grader.
- Latency · The waiting time between sending a request to an AI model and receiving its answer, from the first word through to the last.
- Price per million tokens · How AI APIs bill usage: one rate per million tokens the model reads as input, and another, usually higher, per million it generates.
- Agent orchestration · Coordinating several model calls, tools or AI agents: who does what, in which order, and how the results are brought together.
- Guardrails · Checks placed around an AI model to filter what goes in and what comes out, and to block unwanted answers or actions.
Try it
Full glossary
Frequently asked questions
How is it different from classic monitoring?
Classic monitoring mainly tracks uptime, errors and response times. LLM observability adds the content: prompts, responses, retrieved documents, an agent's intermediate steps and token usage, so you can understand why an answer was bad.
What is a trace?
A trace is the full record of one request, broken into steps called spans: a model call, a document search, a tool call. Each step keeps its inputs, outputs, duration and links to the others.
Is there a standard?
OpenTelemetry is developing semantic conventions for generative AI that define shared names for attributes such as the requested model or the number of input tokens. As of October 22, 2026, these conventions are still marked as in development.
Does logging prompts raise privacy issues?
It can: traces often contain personal data typed in by users. Plan for redaction, limited retention and access control, in line with privacy laws such as the GDPR. This is general information, not legal advice.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.