AI agent monitoring: traces, logs, and alerts

Monitoring an AI agent in production means connecting its actions, errors, response times, and spending to the outcome it actually delivers. Logs record events, traces reconstruct the steps in a task, and metrics trigger alerts when service deteriorates. Useful monitoring also checks output quality and the effects of actions without unnecessarily collecting sensitive data.

Start with the business outcome, not the dashboard

An agent can complete a run without a technical error and still fail at its job. A model call succeeds, a tool accepts the request, and the final response claims an outcome that the business system does not confirm. Monitoring API availability alone would miss this failure.

Define what successful task completion means before choosing your charts. For a ticket-handling agent, success might mean that a proposed resolution has been correctly saved and validated under the applicable rules. For a document research agent, it might mean an answer supported by accessible, relevant sources.

Distinguish between successful completion, technical failure, expected refusal, handoff to a person, abandonment, and an undetermined outcome. A refusal that respects permissions is not necessarily a service failure. Conversely, a final message saying that everything is done does not prove success.

LLM observability helps explain these outcomes through recorded operations. It complements the guide to evaluating an AI agent, which covers test datasets and quality criteria in more detail.

Also specify who needs each type of information. Operations staff need to identify stalled tasks; the business owner needs to know about incomplete outcomes; the person responsible for security needs to investigate sensitive actions. A single dashboard packed with every available data point rarely serves all these needs well.

Connect logs, traces, metrics, and audit records

These signals answer different questions. Keeping them in the same platform can make investigation easier, but their roles remain distinct.

Signal	What it represents	Main use	Limitation
Log	A timestamped event with context	Investigate an error or state change	The sequence of events may remain unclear
Trace	Operations connected to a run	Identify the step responsible for a delay or failure	Incomplete instrumentation leaves blind spots
Metric	A measurement aggregated over time	Detect changes and compare periods	It hides the details of individual cases
Audit log	A sensitive action and its authorization context	Establish who was allowed to act and what effect was observed	It requires appropriate access and integrity controls

Within a trace, a bounded operation is called a span. It has a start, an end, and relationships with other operations. The tracing documentation explains this model and how operations relate to one another.

Use a run identifier to move between signals. An alert should let you open representative traces and then their associated logs without manually searching through an entire conversation.

Keep the business identifier separate from the technical identifier, however. The same request may involve several attempts, resumptions, or deferred processing steps. Treating these as interchangeable would distort statistics and make repeated actions harder to investigate.

Instrument the whole task and each relevant action

Start with an operation that covers the entire task. Then add operations for model calls, document retrieval, validation, and each tool call. The aim is to reconstruct the path taken and its dependencies without turning every internal function into a recorded event.

For each run, capture at least:

The task, attempt, and trace identifiers.
The environment and deployment revision.
The instruction and configuration revisions.
The requested model and, when available, the model actually used.
Timestamps, final status, and stop reason.
Reported resource usage and references to effects in business systems.

For a tool, distinguish the requested action, the authorization check, the outgoing request, and the observed result. A write accepted by an API may still need verification in the destination system before you can treat the action as complete.

The GenAI attributes documented by OpenTelemetry provide a vocabulary for describing models, usage, and operations, among other details. Check which attributes your instrumentation supports before building dashboards that depend on specific fields.

Pay particular attention to boundaries: queues, separate processes, subagents, and remote services. Pass correlation context through a mechanism you control. A resumed task may create a new trace, but it should retain an explicit link to the original task.

Do not try to infer internal reasoning from commentary generated by the model. The evidence you can verify consists of authorized inputs, calls, results, and observed effects.

Measure availability, timing, costs, and quality

Organize indicators around operational questions: whether the service accepts requests, finishes them on time, produces correct results, and stays within its resource budget.

For timing, separate the wait before processing, execution time, and time until a usable result is available. A quick initial response can hide a long delay before the requested action happens. Examine the slowest cases as well: an average can conceal deterioration affecting particular types of tasks.

For reliability, track final failures, retries, tasks with no final status, and unavailable dependencies. Give every rate an explicit denominator. Received, started, and completed tasks represent different populations, so switching between them changes what the metric means.

For costs, include calls across all attempts, services billed by usage, and automated evaluations. Calculate both cost per completed task and cost per successful task. A workflow that is inexpensive per call can become costly when it repeatedly retries.

Finally, distinguish measured quality from a signal that covers only part of it:

Schema validation confirms structure.
A business check confirms an expected state.
Human review assesses an outcome against a rubric.
An evaluator model produces an estimate that can be wrong.

Store the evaluation method and its revision with every assessment. A change to the rubric can shift scores even when the agent’s behavior has not changed.

Build alerts that lead to action

A useful alert identifies a problem, its impact, an owner, and a response procedure. A notification that reports only rising usage, without identifying the affected scope, leaves the recipient to perform the entire diagnosis.

Alerting best practices recommend focusing on symptoms that affect users and avoiding notifications that offer no actionable response. For an agent, that means watching unfinished tasks, invalid outcomes, and unexpected spending increases alongside technical exceptions.

Situation	Signal to watch	Planned response
Requests piling up	Age of queued tasks and total completion time	Check capacity and limit new admissions
Loop with no progress	Repeated actions without useful state changes	Stop the task and retain diagnostic evidence
Abnormal spending	Cumulative usage and spending rate	Enforce a cap or suspend the affected scope
Unexpected sensitive action	Missing authorization or an incompatible effect	Block the affected writes and open an incident
Declining output quality	Business failures and consistent evaluation findings	Investigate the latest change and reduce exposure

Set thresholds based on service commitments, normal operating patterns, and acceptable risk. The same time limit will not suit both an interactive conversation and a deferred document-processing task.

Group alerts that relate to the same incident. Use a confirmation period for ordinary fluctuations while responding immediately to events that require intervention.

An alert does not replace a guardrail. Spending caps, step limits, and write authorization must be enforced during execution, regardless of whether someone receives a notification.

Choose a platform based on the information you lack

A specialized platform makes execution paths easier to inspect and connect to evaluations. It should complement monitoring for your servers, queues, and business services. The AI tools directory places these solutions alongside other operational tools.

The following capabilities are documented in the official sources consulted for this page. The table is not a performance ranking.

Solution	Execution paths and traces	Evaluation	What to check for your use case
Langfuse	Nested observations, usage, and timing	Scores, datasets, and experiments	Permitted data collection, hosting, and project organization
LangSmith	Agent traces and production monitoring	Online evaluations and execution analysis	Supported integrations and deployment requirements
Arize Phoenix	Instrumentation based on OpenTelemetry and OpenInference	Datasets, evaluations, and experiments	Operating the instance and instrumentation compatibility

The Langfuse documentation, official LangSmith Observability overview, and official Phoenix repository describe these features. They do not establish whether the products suit your workload.

Compare the options using a representative workflow that includes a parallel branch, resumption after an interruption, a failed call, and masked content. Also check data export and whether you can delete an individual run.

Total cost includes ingestion, storage, retention, evaluations, and operations. An entry-level subscription price is therefore insufficient for comparing two architectures.

Limit collection and control data retention

A trace can contain a private request, a retrieved document, a tool response, or authentication information. It creates an additional copy of data that was previously spread across several services.

Define an allowlist of fields. Start by collecting the metadata needed for diagnosis, and enable content collection only for an explicit need. A document reference, its revision, and a check result may be sufficient without copying its full text.

Apply filtering before export. The Langfuse masking documentation describes this capability and explains that processing by one exporter does not necessarily sanitize data sent through another.

Check exceptions, attachments, operation names, and fallback logs as well. Masking the main content field does not protect a copy stored elsewhere.

Separate access by environment and responsibility. Set retention periods according to purpose: routine diagnosis, audits of sensitive actions, and examples used for evaluations do not necessarily need the same retention period. Self-hosting leaves these decisions in your hands.

The guide to securing an AI agent covers permissions and secrets. The article on AI agents and personal data complements this inventory of data flows.

Roll out monitoring in verifiable stages

The prerequisites are organizational as well as technical: a service owner, a definition of success, a test environment, and controlled access to business outcomes. Without them, you will collect data without being able to resolve incidents.

Define states and identifiers. Document possible outcomes, distinguish tasks from attempts, and specify how events should be linked.
Trace a complete workflow. Capture receipt, queue time, calls, checks, effects, and final status. Verify what happens during an interruption as well.
Add essential measurements. Volume, timing, outcomes, retries, and usage should be traceable to the runs behind them.
Apply collection rules. Use recognizable synthetic data to check what actually reaches each destination.
Trigger alerts in testing. Simulate an unavailable dependency, a stalled task, and a reached cap. Check both notification delivery and the response procedure.
Introduce traffic gradually. Compare the exposed scope with a reference workflow and keep a rollback option available.

Also test collector failure. An observability platform outage should not accidentally block the entire service. For actions that your policy requires you to audit, explicitly decide whether they remain permitted when recording fails.

Measure collection overhead, including memory use, export queues, and latency. Asynchronous export reduces coupling, but instrumentation still consumes resources.

Sample without losing incident evidence

Retaining all content from every run increases both storage volume and data exposure. Sampling can limit detailed traces, provided you understand which conclusions the resulting data can support.

You can select a portion of ordinary runs and apply a separate policy to errors or sensitive workflows. Selection based on the final outcome, however, requires temporarily retaining enough information to make that decision.

Do not calculate the overall failure rate from a dataset deliberately weighted toward errors. Use counters covering all activity for general metrics, then use selected traces to explain individual cases.

Monitoring should also provide evidence that it is working. Track ingestion rejections, export failures, dropped data, and arrival delays. A decline in trace volume may indicate lower traffic, or it may indicate a collection failure.

Finally, do not let routine sampling discard audit events that your policy requires you to retain. Diagnostic traces and evidence of sensitive actions need separate rules.

Common problems and incident response

Calls appear without links between them. Look for lost context at a queue or process boundary. Also check whether parallel tasks are reusing context that is no longer correct.

Usage figures are incomplete. Distinguish unavailable values from zero usage. Investigate retries, interrupted streaming responses, and calls made outside the main instrumentation.

The dashboard reports success, but the action is missing. Replace status derived from the final response text with a check of the destination system’s state. Store the agent’s claim separately from the observed result.

Traces contain duplicate operations. Several layers may be instrumenting the same call. Identify which layer creates each operation before adding up durations or usage.

During an incident, first contain the effects: suspend affected writes, stop loops, or redirect requests. Then preserve relevant traces and configuration revisions so the investigation has a reliable record.

Build a timeline connecting the request, authorization, action, and final state. Do not directly replay a trace that could send a message or modify a record. After fixing the issue, turn the case into a redacted regression test and verify that the alert would have detected the problem.

Checklist before going live

Success is verified in the business system.
Each attempt is linked to its task and operations.
Retries, queue time, and interruptions remain visible.
Unknown costs are distinguished from zero costs.
Sensitive content is filtered before every export.
Alerts identify an owner and an action.
Caps and stop mechanisms work independently of notifications.
Collection failures and rollback have been tested.
Relevant incidents feed into regression tests.

Frequently asked questions

What is the difference between monitoring and observability for an AI agent?

Monitoring checks for conditions you have anticipated, such as a stalled task or an exceeded budget. Observability also provides the evidence needed to investigate unexpected behavior. Both support day-to-day operations.

Should you log every prompt in production?

No. You can start with metadata, durations, statuses, and usage, then allow targeted collection of redacted content. The decision depends on the data you handle and what you need to diagnose.

Can you monitor an agent without a specialized platform?

Yes, using structured logs, metrics, and traces in your existing infrastructure. A specialized platform mainly makes model interactions easier to inspect and connect to evaluations. First, identify which information you actually lack.

Can a trace reproduce a run exactly?

A trace documents the operations it captured, but it does not guarantee an identical replay. The model, retrieved data, and external service state may have changed. Replaying a write operation therefore requires an isolated environment with external side effects disabled.

How do you measure an agent's quality after deployment?

Combine checks on business outcomes, human review of selected runs, and automated evaluations. Keep an undetermined category for cases where the evidence is insufficient. An automated score should always include its method and limitations.

Monitoring an AI agent in production: logs, traces, alerts, and observability tools