LLM Observability: How to Monitor, Trace, and Debug AI Apps and Agents in Production

A request may complete successfully, but return the incorrect response. A RAG system may retrieve documents that are not relevant to the question. An agent may call the wrong tool or repeat an action. Standard monitoring can confirm the request was fulfilled, but it can't always explain why the output is wrong.

Developers can examine traces to see which steps were completed and where something went wrong. They can also monitor response time, token usage, and cost. Response evaluations help them check whether the answers are accurate and meet the application's requirements.

This guide covers the key metrics to monitor, how tracing applies to RAG systems and agents, and the available tools. It also covers how to enable monitoring on production applications.

What Is LLM Observability?

It is the process of tracking what happens inside a language model application when it receives a request. It uses traces, logs, and metrics to record model calls, tool responses, and other steps taken before an answer is returned. Developers use this information to understand how the application works and investigate problems.

For example, a customer asks a chatbot about the company's refund policy and receives the wrong answer. The problem could be an outdated document retrieved from the knowledge base or incorrect information returned by a connected service. By checking the trace, developers see which information reached the model and where the problem started.

These records also help find problems with response time and cost. A request might take longer because the application waits for a slow tool or makes repeated model calls. Token usage might increase because too much information is sent with each request. With this information, developers see more than the final response. They know what happened during the request, how long each step took, and which part needs attention.

How It Differs from Traditional DevOps Observability

Traditional DevOps tools monitor how an application runs. They indicate whether requests are failing, servers are overloaded, or if a service is taking too long to respond. Developers use this information to find and fix technical problems.

Language model applications can return wrong answers without reporting any errors. A chatbot may answer a question using an unrelated document. An agent can still choose the wrong tool even after a tool call succeeds.

DevOps monitoring can show that a request was completed and how long it took. It may not reflect what information is fed into the model, or what occurs from one step to the next. To investigate a wrong answer, developers need to view the prompt, retrieved documents, model response, and tool calls.

The two methods are used for different aspects of the same application. Traditional tools monitor the software and infrastructure. LLM-specific tracing provides details on model invocations and the operations performed to generate a response.

Why LLM Applications Need Dedicated Observability

Language models do not always give the same answer to a question. An application may work well during testing but give wrong answers when people start using it. These requests can finish without an error, even when the information in the response is incorrect.

Consider a customer support chatbot that uses company documents to answer questions. A customer asks about refunds, but the system retrieves an old refund policy. The chatbot gives an incorrect answer even though the API works normally. Developers need to see which document was retrieved to find the problem.

The cost of running the application can also change from one request to another. A short question may use thousands of tokens if the prompt includes a long conversation history or too many documents. If this happens often, the business may spend more on model usage than expected.

Agents can be harder to debug because they perform several actions to complete a task. An agent may call the wrong tool or keep repeating a failed action. Developers need to see each tool call and its response to understand what went wrong.

There are security risks too. Someone may enter private information into a prompt or try to make the model ignore its instructions. The application may process the request without reporting an error. Developers need to check these interactions for possible security issues.

Regular monitoring does not provide all this information. Teams need records of model calls, retrieved documents, tool responses, and token usage. They can use these records to investigate incorrect answers and find problems that normal application logs may miss.

LLM Observability vs LLM Monitoring: What’s the Difference?

Monitoring allows developers to know how an application is doing. They can check response times, failed requests, token usage, and costs. They can also configure notifications for unexpected changes. Observability provides the records they need to investigate those changes and see what happened during a request.

For example, a customer support chatbot may give a customer the wrong refund policy. The request finishes successfully, so the error rate does not change. A quality check may identify the incorrect answer. Developers can then review the trace to see whether the chatbot retrieved an outdated policy or received incorrect information.

Another example is token usage. Monitoring may show that the application is using more tokens than usual. The team can check individual requests to find out why. The app might be passing too much of the conversation history or calling the model more times than it should.

Security issues could also be overlooked. A user might include instructions in a prompt that try to make the model ignore its rules. Similar instructions can come from a retrieved document. These can be identified through security checks, and interaction information can be displayed to the developer so they can observe how the model reacted. These records should be stored with appropriate protection for sensitive data.

The monitoring system is used to identify issues throughout the application. Observability lets them examine the requests involved and find the cause. Developers use both when maintaining language model applications in production.

Core LLM Observability Metrics to Monitor

After an application goes live, developers need to check how it performs with real users. Some requests may take longer than expected or use too many tokens. Others may return incorrect answers or make unnecessary tool calls. The metrics below help you find these problems:

Traces and Spans

When someone sends a request, the application may go through several steps before returning an answer. A trace records these steps so developers can see what happened during the request.

A RAG application may follow this process:

User Question → Document Retrieval → Model Call → Response

Each recorded step is called a span. It contains details about one operation, such as how long it took, what information was used, and whether an error occurred.

For example, a customer asks about the return policy. The application searches the company knowledge base and sends the selected document to the model. If the answer is incorrect, developers can check the retrieval span to see which document was used.

If the application selected an old policy, the problem may be in the retrieval step. If it selected the correct document, developers can check what was sent to the model and how it answered.

Agent requests may involve several model calls and tools. A trace keeps a record of these steps, including repeated calls or failed actions. Developers can follow the request and check where the problem occurred.

Quality Signals: Groundedness, Relevance

A response may sound correct even when it contains wrong information. Groundedness and relevance help check the quality of answers produced by a language model.

Groundedness checks whether the answer matches the information provided to the model.

For example, a company document says customers can return an item within 14 days. The chatbot tells a customer they have 30 days. The answer is not supported by the document.

Relevance checks whether the answer addresses the user's question.

A customer asks how to cancel an order, but the chatbot explains how to change an order. The information may be correct, but it does not answer the question.

These checks can catch incorrect or unrelated answers even when the request finishes without an error.

Latency (Time to First Token and End-to-End)

Latency measures how long an application takes to respond. Two measurements help show where users experience delays.

Time to First Token (TTFT) measures how long a model takes to start returning its response after receiving a request. A high TTFT means users may have to wait several seconds before the answer starts appearing.

End-to-end latency measures the time from when a user sends a request until the complete response is returned. It includes document retrieval, model calls, and any other steps the application performs.

For example, a RAG application may take longer because document retrieval is slow. An agent may spend several seconds waiting for an external tool. Checking the time spent on each step can help developers find what is causing the delay.

Token Usage and Cost per Request, User, and Feature

The cost of using a language model depends on the model's pricing and how many tokens it processes. Input tokens include the prompt, conversation history, and documents sent to the model. The output tokens include the model's output.

Even a short question can require thousands of tokens if the application includes a long conversation history or multiple large documents. Agents may use even more because they often make multiple model calls to complete one task. Track token usage and cost for each request. You can also view the total cost of various users or features over time.

If a feature starts costing more than expected, review its prompts and model calls. The application might be sending unnecessary data or making unnecessary calls.

Safety Signals: PII Leakage, Jailbreaks, Prompt Injection

Language model applications can reveal private information or accept instructions that bypass security rules. Even if the request is successful, these risks should be reviewed.

PII leakage happens when personal information is shared with someone who should not see it. For instance, a chatbot might display a customer's email address or account information to another customer.

A jailbreak is an attempt to make the model ignore its restrictions. Someone may use a specially written prompt to get a response the model would normally refuse.

Prompt injection happens when instructions from an untrusted source try to change the model's behavior. These instructions can be found in documents, on websites, or in tool responses. A RAG system can return a document that will cause the model to disregard its instructions.

Developers should log any unusual activity and determine if the model executed the unwanted behavior. Sometimes an attempted attack is not a successful attack. Traces can show what the model was given and how it reacted.

Personal information can also be included in prompts and traces. Limit who can access these records, and avoid storing private data that isn't needed.

User Feedback and Drift

Users may find issues that automated tests miss. A thumbs-down rating or a reported answer can guide developers to answers that need review.

If possible, preserve user feedback with the original request. If multiple customers report errors in their refunds, the team can review refund requests to see whether the same document or prompt was the issue.

The quality of the response may also vary over time. This is called drift. Customers may start asking different questions, the application may use different documents, or a model change may affect the answers.

To identify these changes, compare response quality over time. If more answers become incorrect, compare the requests before and after the update. This can help determine whether the issue was with the prompt, the model, or the retrieved information.

Observability for RAG Pipelines

A RAG system searches for information before the model generates an answer. It finds relevant documents and sends their content to the model along with the user's question. If the search yields incorrect information, then the answer may also be incorrect.

Suppose an employee asks about the company's leave policy. The system identifies an old policy rather than the most current policy. The model uses that document and gives an incorrect answer. The request completes successfully, and the employee receives outdated information.

The trace can show which documents were retrieved and which were sent to the model. Verify that the search returned the right document and that it appeared high enough in the results to be included. Also check that the information is current.

The model can still be wrong even when the correct document is fed into it. It may leave out an important detail or give an answer that does not match the source. Groundedness checks can help find these problems.

Retrieval also takes time and adds to a request's cost. Searching documents can slow down the response. Large documents also use more tokens when sent to the model. Tracking retrieval time and token usage helps find where the extra time and cost come from.

If a RAG system gives a wrong answer, check the retrieved documents first. If the right information was sent to the model, review the prompt and response to find what went wrong.

AI Agent Observability: Tracing Multi-Step Workflows

An agent can make several model calls or use different tools to complete a task. It may also send part of the work to another agent. The steps can change depending on the request and the information returned by each tool.

A trace records these steps, including which tools were used and what they returned. Developers can review the record if a task fails or the agent takes an unexpected action.

How Agents Are Harder to Observe

A chatbot may answer a question with a single model call. An agent works differently because it can take several actions before completing a request. Each action may depend on the previous result.

Suppose a customer asks an agent to cancel an order. The agent first calls the order service to check its status. If the order has not been shipped, it asks the customer to confirm the cancellation. Once the customer confirms, the agent sends a cancellation request to the order system. It then checks the result before telling the customer whether the order was canceled.

The task may not always go as planned. The order service could return an old status, or the cancellation request could fail. If the agent does not receive a clear response, it might send the same request again. The customer may receive a final message without knowing that something went wrong during the process.

A trace shows the order of these actions. Developers can check the order status returned by the first tool, the customer's confirmation, the cancellation request, and the response from the order system. If the agent repeated a request or acted on incorrect information, they can see where it happened.

What to Capture: Tool Calls, Reasoning Steps, Loops, Handoffs

An agent may take several steps to finish a task. The trace should show what it did at each step, how long it took, and what happened next. Model calls and tool calls can be recorded as separate spans.

Tool calls show which tool the agent used, what it sent, and what the tool returned. For an order cancellation, the trace may show the order status check followed by the cancellation request. Developers can check whether both calls worked.

Reasoning steps show the decisions made during the workflow. An agent may check an order's status before deciding whether to request approval for cancellation. The trace can record these actions without storing the model's private internal reasoning.

Loops occur when an agent repeats an action. It may retry a tool call after a failure. But if it keeps making the same call without getting a useful result, the task may be stuck. Recording each attempt helps developers find repeated calls that waste time and tokens.

Handoffs happen when one agent passes a task to another. A customer support agent may send a refund request to a payment agent. The trace should show what information was passed and what the second agent did with it.

All these steps should be linked to the same request. Developers can then check the complete sequence to find where a task failed. Personal information and other sensitive data should be removed or protected before storing the traces.

Standardizing with OpenTelemetry GenAI Semantic Conventions

Different models and agent frameworks may use different names to record the same action. This can make it hard to compare traces or collect them in one place.

OpenTelemetry helps record this information in a common format. Its GenAI semantic conventions define standard names for model calls, agent actions, and other operations.

For example, execute_tool is used when an agent runs a tool, while invoke_agent is used when an agent is called. The GenAI span conventions explain how to record these actions in a trace.

Each span can also include operation details. The GenAI attribute registry defines fields for the model name and the number of input and output tokens.

Developers can use these conventions to collect traces from different models and frameworks in the same format. This makes the records easier to compare and review.

OpenTelemetry does not check whether an agent made the right decision or completed a task correctly. Developers still need tests and evaluations for that.

Some GenAI conventions are still being updated. Check which version your tracing library supports before using them in production.

Best Practices for AI Agent Observability

Agent traces should show what happened during a task. Missing steps can make it hard to find errors, while too much data can make the records harder to read.

Here are some practices to follow when monitoring agents in production.

1. Keep Every Agent Step in One Trace

An agent may make several model calls and use different tools to complete one task. Recording these calls separately makes it hard to see how the task was completed.

Keep all the steps in one trace. For example, if an agent checks an order, requests approval, and then cancels it, the trace should include each action and its result. If the cancellation fails, developers can check what happened at each step.

The OpenTelemetry documentation explains how model calls and tool operations can be recorded as separate spans within the same trace.

2. Decide What a Successful Task Looks Like

An agent might tell a customer that their order has been canceled even though the order is still active in the system. The request appears complete, but the actual work has not been done.

Check the result before marking a task as successful. For an order cancellation, verify that the order status has changed in the database. For a support request, check whether the customer's issue was resolved. The agent's final response should match what actually happened in the system.

3. Watch for Repeated Actions and Failed Tool Calls

An agent sometimes needs to retry an action when a tool fails or does not respond. For instance, an order service is temporarily unavailable. The agent tries again, but the service keeps returning an error. Repeating the same request will only waste time if the problem continues.

Track failed tool calls and the number of retries for each task. Set a limit on how many times the agent should retry and how long it should keep working. If the tool still fails, the agent should stop and report the problem instead of repeating the action.

Repeated model calls also increase token usage and cost. Check the total time and tokens used for each task to find agents that spend too much time retrying failed actions.

4. Check Agent Quality with Regular Evaluations

An agent may complete a task but miss an important step. Traces can show what the agent did, while evaluations help check whether it followed the correct process and completed the work properly.

For example, a refund agent must check whether an order qualifies for a refund before sending the request to the payment system. If it processes the refund without checking the order, the task may appear successful even though the agent did not follow company policy. Regular evaluations can help find these mistakes.

You can also use failed production requests to create test cases. If an agent processes a refund incorrectly, save that case and test it again after making changes. This helps developers confirm the problem is fixed and that the agent follows the required steps. LangChain's guide to production agent monitoring explains how teams can review traces and use evaluations to check agent behavior after deployment.

5. Protect Sensitive Information in Traces

Agent traces often contain customer details or information from internal systems. Storing complete prompts and tool responses could expose private data to people who should not have access to it.

Before saving traces, remove any sensitive information that is not needed for debugging. If a payment fails, developers need the error message and request ID to investigate the issue, not the customer's payment details.

Access to these records should be limited to authorized employees. The team should also decide how long to keep them and delete records that are no longer needed.

6. Set Alerts for Problems That Need Attention

Developers cannot review every agent request manually. They need alerts when something starts going wrong. For example, a tool might fail repeatedly, or an agent might take much longer than usual to finish a task.

Set limits for tool failures, execution time, and token usage. If an agent keeps calling the same tool, the extra calls will increase both the time and cost of the request. An alert will help the team notice the problem.

Each alert should have a link to the related trace so developers can see what happened. Keep the number of alerts low at first. Once you have enough production data, adjust the limits and remove alerts that keep reporting minor issues.

LLM Observability Tools to Consider

The tools used to monitor language model applications offer different features. Some provide detailed traces and response evaluations. Others bring model activity into an existing monitoring system. The choice depends on how the application is built and what the development team needs to check.

Here are four tools used for this work.

Langfuse is an open-source platform for tracing and evaluating language model applications. It records model calls, response times, and token usage. Developers also use it to manage prompts and test responses after making changes to the application. Langfuse offers a cloud version and a self-hosting option for companies that want to run it on their own infrastructure.

Arize Phoenix is an open-source tool for tracing requests and evaluating responses. It is particularly useful for RAG systems. If a chatbot gives the wrong answer, developers can review the retrieved documents and check what information was sent to the model. Phoenix also lets you run evaluations and compare results. Further information is provided in the Phoenix documentation.

Datadog LLM Observability brings model monitoring into Datadog's existing platform. It logs model calls, agent activity, response times, and token consumption. A company already using Datadog for application monitoring would have these records alongside its other system metrics and logs. Datadog's documentation details its capabilities and the integrations it supports.

LangSmith is used for tracing and testing language model applications. It allows developers to view each request and store examples for future testing. For instance, if a bot provides the wrong response, the team can save this inquiry and retest when the prompt is updated. LangSmith supports LangChain and LangGraph, along with applications built using other frameworks. Its evaluation guide explains how to set up these tests.

Open-Source vs Managed Platforms: Which Is the Right Approach?

The difference between self-hosted and managed platforms depends on who manages and runs the software. Self-hosting involves deploying the monitoring tool on your company's own servers or cloud resources. You decide where the traces are stored. Langfuse and Arize Phoenix both offer this option.

Self-hosting involves more work. Your team must install the software, update it, and ensure that it functions correctly. Storage is another expense. The more people use the application, the more traces are collected, and the more space is required to store them.

A managed platform means the software is running on the provider's platform. Your developers connect the application and set up tracing without managing servers. You pay based on the provider's pricing. Langfuse offers both cloud and self-hosted versions. LangSmith also offers self-hosted and hybrid options for enterprise customers.

Before you pick and choose, find out how the platform stores data. Traces might contain customer messages or information from internal systems. Know where these records are stored, who can access them, and when they will be destroyed. If your company needs to store data in a specific region, ensure that the platform does the same.

Evaluation Criteria

Choose a monitoring tool based on what your application needs. Check whether it works with your existing setup and provides the information your developers need to investigate problems. Cost and data storage requirements also affect the decision.

OpenTelemetry Support

If your application is already using OpenTelemetry, verify that the platform supports it. This enables developers to trace a request from the main application through model calls and connected services.

Arize Phoenix uses OpenTelemetry for tracing. Datadog also supports OpenTelemetry for monitoring language model applications. Review the platform's available tracing properties and versions before selecting it. Not all tools have access to the same data and integrations.

Built-In Evaluations

Traces provide information about the activity that has taken place in the request, but you should also verify that the application returned the expected results.

In a RAG system, this involves verifying if the retrieved documents align with the question and provide the answer. For an agent, it implies confirming that the right tool was applied and the work was done.

Review the evaluations offered by the platform. Your team should also be able to create tests for its own use cases and run them on production requests.

Self-Hosting and Data Residency

Traces may include customer messages or private company information from company systems. Check how the platform protects this data before storing it.

Ask if sensitive information is removed or hidden, who can access the records, and how long they are retained. If your company must store data in a specific country or region, check the available storage locations. Self-hosting provides greater control of data storage, but your team must maintain the platform.

Pricing

Monitoring platforms charge for usage in different ways. Some charge based on recorded spans, while others offer plans with fixed usage limits. For example, Datadog's Agent Observability pricing is based on LLM spans. An agent that makes several model calls during one task generates multiple billable LLM spans. This pricing model does not include tool and retrieval spans.

LangSmith and Langfuse also publish their plans and usage limits. Compare these limits with the amount of tracing data your application produces. Self-hosting also has costs. Even if the software is free, you will still have to pay for servers, storage, and maintenance.

To estimate the cost, check how many requests your application receives each month and how many model calls each request makes. Compare pricing plans using these numbers and calculate the additional monitoring costs when there's a usage increase.

How to Implement LLM Observability: Step-by-Step

Setting up LLM observability depends on how your application works and what you need to monitor. A chatbot, RAG system, and agent perform different tasks, so the information worth tracking will also be different. Start with the metrics that help your team check performance and find problems. Add more measurements as the application grows.

Define Success Metrics per Use Case

Before setting up monitoring, decide what a successful request means for your application. For a RAG system, the search should find relevant documents, and the answer should match the information in those documents. You can assess this using retrieval relevance and groundedness.

An agent needs different checks because its job might require an action in another system. For example, a customer service agent that cancels orders should update the order status in the database. You can review task and tool failure statuses to determine whether the agent completed the task or the tool failed.

You also have to determine what constitutes an unsuccessful request. The system should not be considered successful if the wrong document is used in a RAG system, or if an agent confirms an order cancellation without updating the database. These results let an application developer know which requests require additional review.

Instrument the Application

The next step is to add tracing to your application. Each user request gets a trace ID, and the main steps are recorded as separate spans. For a RAG system, these steps could be document retrieval and model calls. An agent might also use external tools or pass work to another agent.

Record the time taken by each step and whether it succeeded or failed. For model calls, store the model name and the number of tokens used. Link all these records to the same trace ID so developers have the full history of a request when investigating a problem.

If you already use OpenTelemetry, you do not need a separate tracing system for language model workflows. The OpenTelemetry GenAI semantic conventions define common fields for recording model calls and agent operations.

Before storing traces, check whether they contain customer details or private company information. Remove or mask any sensitive data developers don't need for debugging.

Build Dashboards and Alerts

Once tracing is set up, use dashboards to check how the application performs over time. Track response times, token usage, and costs. For a RAG system, answer quality is also worth checking. An agent may need additional metrics, such as failed tool calls and task completion rates.

The dashboard should show the information your team needs to monitor regularly. Adding too many metrics makes it harder to find problems. Keep the ones that help developers understand whether the application is working properly.

Set alerts when response times, tool failures, or token usage go above normal levels. Use production data to decide these limits. For example, a five-second response might be acceptable for an agent completing several steps but too slow for a simple chatbot. The limits should match how your application normally performs.

Connect Production Traces to Evaluations

Production traces show how the application works with real users. You can use these records to find problems and create tests based on requests that failed.

For example, several customers get incorrect answers about refunds. Developers review the traces and find that the system retrieved an old refund policy. They save these requests as test cases and update the retrieval system.

After making the changes, they run the same tests to check whether the system now finds the correct policy and gives the right answer. These tests should also be run after future updates to make sure the problem does not return.

Over time, the team builds a collection of test cases from real user requests. This helps developers check how changes to prompts, models, or retrieval settings affect answers that previously caused problems.

Building Production-Ready AI with Coding Crafts

We help businesses build language model applications with proper tracing and monitoring. Our developers look at how the application works and decide what needs to be tracked. For a RAG system, this could mean checking which documents were retrieved and whether the answers match the source. For an agent, it could mean tracking tool calls and checking whether tasks were completed.

If you already have a working application, we review its current monitoring setup and identify what is missing. For new projects, we plan these requirements during development, so your team has the information needed to find and fix problems after launch.

Explore our Custom AI Development Services or contact Coding Crafts to discuss your project.