# Dec 15th, 2025: \[EN\] AI Agent Observability with OpenLit, OpenTelemetry and Elastic

**URL:** <https://discuss.elastic.co/t/dec-15th-2025-en-ai-agent-observability-with-openlit-opentelemetry-and-elastic/383341>\
**Category:** Advent Calendar\
**Created:** [December 15, 2025, 8:00am UTC](https://discuss.elastic.co/t/dec-15th-2025-en-ai-agent-observability-with-openlit-opentelemetry-and-elastic/383341 "2025-12-15T08:00:25Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [December 15, 2025, 8:00am UTC](https://discuss.elastic.co/t/dec-15th-2025-en-ai-agent-observability-with-openlit-opentelemetry-and-elastic/383341/1 "2025-12-15T08:00:25Z")

</div>

![social-advent-2021-Day15](https://us1.discourse-cdn.com/elastic/original/3X/b/8/b89fed9ace86e28707098e3ac1f698693973dade.png)

Traditionally, we have relied on Observability diagnostic information to understand what our applications are doing. Particularly over the festive season where we live in fear of a ping if we drew the on-call short straw. Now that we're building AI agents, we need a way to observe them too!

Here we'll walk through how to use [OpenLit](https://openlit.io/) and OpenTelemetry to generate telemetry you can use to diagnose issues and identify common issues. Specifically, we'll instrument a simple travel planner (for those seeking warmer climbs this season ☀) built using [AI SDK](https://ai-sdk.dev/) available in [this repo](https://github.com/carlyrichmond/observing-ai-agents):

![Travel Planner Context Eng](https://us1.discourse-cdn.com/elastic/original/3X/3/5/355692023ee2dfd4582d4ebde94f8c04e4f72931.gif)

## What are OpenLit and OpenTelemetry?

[OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) is a CNCF incubation project where SDKs and tools for generating information on how software components are behaving to help you diagnose issues.

[OpenLit](https://openlit.io/) on the other hand is an open source tool that generates OTLP (or OpenTelemetry Protocol) signals to show LLM and vector database interactions within AI agents. It currently has SDK support for Python and TypeScript (of which we'll use the latter) and a Kubernetes operator.

## Basic AI tracing & metrics

Before we instrument out AI code, we need to install the required dependencies:

```zsh
npm install openlit

```

Once installed, we need to initialize OpenLit within the segment of the application where we invoke the LLM. In our application, this it the `route.ts` file:

```ts
import openlit from "openlit";

import { ollama } from "ollama-ai-provider-v2";
import { streamText, stepCountIs, convertToModelMessages, ModelMessage } from "ai";
import { NextResponse } from "next/server";

import { weatherTool } from "@/app/ai/weather.tool";
import { fcdoTool } from "@/app/ai/fcdo.tool";
import { flightTool } from "@/app/ai/flights.tool";
import { getSimilarMessages, persistMessage } from "@/app/util/elasticsearch";

// Allow streaming responses up to 30 seconds to address typically longer responses from LLMs
export const maxDuration = 30;

const tools = {
  flights: flightTool,
  weather: weatherTool,
  fcdo: fcdoTool,
};

openlit.init({
  applicationName: "ai-travel-agent", // Unique service name
  environment: "development", // Environment (optional)
  otlpEndpoint: process.env.PROXY_ENDPOINT, // OTLP endpoint
  disableBatch: true /// Live stream for demo purposes
});

// Post request handler
export async function POST(req: Request) {
  const { messages, } = await req.json();

  // Memory persistence omitted

  try {
    const convertedMessages = convertToModelMessages(messages);
    
    const prompt = `You are a helpful assistant that returns travel itineraries based on location,
     the FCDO guidance from the specified tool, and the weather captured from the
     displayWeather tool.
     Use the flight information from tool getFlights only to recommend possible flights in the
     itinerary.
     If there are no flights available generate a sample itinerary and advise them to contact a
     travel agent.
    Return an itinerary of sites to see and things to do based on the weather.
    If the FCDO tool warns against travel DO NOT generate an itinerary.`;

    const result = streamText({
      model: ollama("qwen3:8b"),
      system: prompt,
      messages: allMessages,
      stopWhen: stepCountIs(2),
      tools,
      experimental_telemetry: { isEnabled: true }, // Allows OpenLit to pick up tracing
   );

    // Return data stream to allow the useChat hook to handle the results as they are streamed through for a better user experience
    return result.toUIMessageStreamResponse();
  } catch (e) {
    console.error(e);
    return new NextResponse(
      "Unable to generate a plan. Please try again later!"
    );
  }
}

```

To send our traces to Elastic, we need to specify the OTLP endpoint as part of our OpenLit configuration which sends to a local OpenLit instance and the console by default. Given this is a frontend client, it's best practice to send the signals via a proxy and collector (as discussed in [this piece on frontend tracing in Observability Labs](https://www.elastic.co/observability-labs/blog/web-frontend-instrumentation-with-opentelemetry)).

The other quirk that is specific to our AI SDK instrumentation is that we need to enable the generation of telemetry using `experimental_telemetry: { isEnabled: true }`. With both sets of configuration, OpenLit will generate traces that show the distinct tool calls in our application, as well as key requests to Elasticsearch to persist our semantic memory:

 ![otel-record-store-b48988.kb.eu-west-1.aws.elastic.cloud_app_apm_services_ai-travel-agent_transactions_view_comparisonEnabled=true&environment=ENVIRONMENT_ALL&kuery=&latencyAggregationType=avg&offset=1d&rangeFrom=now-24h%2Fh&rangeTo=now&serv](https://us1.discourse-cdn.com/elastic/original/3X/c/6/c6a70ff105e9eb82e92a2c0e4c6fbaf00a0f9a09.png)

The default instrumentation also generates metrics including input and output token counts which can be used to track usage and identify usage trends:

 ![Screenshot 2025-11-13 at 18.12.07](https://us1.discourse-cdn.com/elastic/original/3X/3/8/38f10843a6f256587a72301d3c64a810a2fee8a2.jpeg)

Of course you can also create custom dashboards to track these counts as well, with the JSON for the below dashboard available [here](https://github.com/carlyrichmond/observing-ai-agents/tree/main/dashboard):

 ![Screenshot 2025-11-13 at 18.12.30](https://us1.discourse-cdn.com/elastic/original/3X/6/f/6fd0895b83b1f71dffa3fbb8af7a1ba4bfe40812.jpeg)

## Evaluating accuracy with Evaluations

[Evaluations](https://docs.openlit.io/latest/openlit/quickstart-evals) use another LLM to evaluate the accuracy of generated responses to identify inaccuracies, biases, hallucinations and toxic content such as mentions for violence. This is achieved via a mechanism known as [LLM as a Judge](https://en.wikipedia.org/wiki/LLM-as-a-Judge). At time of writing OpenLit supports evaluation via [OpenAI](https://openai.com/) and [Anthropic](https://www.anthropic.com/).

It can be configured and triggered via the following configuration:

```typescript
// Imports and tools omitted

openlit.init({
  applicationName: "ai-travel-agent",
  environment: "development",
  otlpEndpoint: process.env.PROXY_ENDPOINT,
  disableBatch: true
});

const evals = openlit.evals.All({
  provider: "openai",
  collectMetrics: true,
  apiKey: process.env.OPENAI_API_KEY,
});

// Post request handler
export async function POST(req: Request) {
  const { messages, id } = await req.json();

  try {
    const convertedMessages = convertToModelMessages(messages);
    const allMessages: ModelMessage[] = previousMessages.concat(convertedMessages);
    
    const prompt = `You are a helpful assistant that returns travel itineraries based on location, the FCDO guidance from the specified tool, and the weather captured from the displayWeather tool.
        Use the flight information from tool getFlights only to recommend possible flights in the itinerary.
        If there are no flights available generate a sample itinerary and advise them to contact a travel agent.
        Return an itinerary of sites to see and things to do based on the weather.
        Reuse and adapt the prior history if one exists in your memory.
        If the FCDO tool warns against travel DO NOT generate an itinerary.`;

    const result = streamText({
      model: ollama("qwen3:8b"),
      system: prompt,
      messages: allMessages,
      stopWhen: stepCountIs(2),
      tools,
      experimental_telemetry: { isEnabled: true },
      onFinish: async ({ text, steps }) => {
        const toolResults = steps.flatMap((step) => {
          return step.content
            .filter((content) => content.type == "tool-result")
            .map((c) => {
              return JSON.stringify(c.output);
            });
        });

       // Evaluate response when received from LLM
        const evalResults = await evals.measure({
          prompt: prompt,
          contexts: allMessages.map(m => { return m.content.toString() }).concat(toolResults),
          text: text,
        });
      },
    });

    // Return data stream to allow the useChat hook to handle the results as they are streamed through for a better user experience
    return result.toUIMessageStreamResponse();
  } catch (e) {
    console.error(e);
    return new NextResponse(
      "Unable to generate a plan. Please try again later!"
    );
  }
}

```

OpenAI will scan the response and flag inaccurate, toxic or biased content, similar to the below:

 ![Screenshot 2025-11-13 at 18.38.14](https://us1.discourse-cdn.com/elastic/original/3X/3/a/3a839eb0d435cfe384f7d5fead9cf6385cd0a57c.jpeg)

## Detecting malicious activity with Guardrails

In addition to evaluations, [Guardrails](https://docs.openlit.io/latest/openlit/quickstart-guard) allow us to monitor that the LLM adheres to content restrictions we place on responses, such as not disclosing financial or personal information. They can be configured via the below code:

```typescript
// Imports omitted

openlit.init({
  applicationName: "ai-travel-agent",
  environment: "development",
  otlpEndpoint: process.env.PROXY_ENDPOINT,
  disableBatch: true
});

const guards = openlit.guard.All({
  provider: "openai",
  collectMetrics: true,
  apiKey: process.env.OPENAI_API_KEY,
  validTopics: ["travel", "culture"],
  invalidTopics: ["finance", "software engineering"],
});

// Post request handler
export async function POST(req: Request) {
  const { messages, id } = await req.json();

  try {
    const convertedMessages = convertToModelMessages(messages);
    const allMessages: ModelMessage[] = previousMessages.concat(convertedMessages);
    
    const prompt = `You are a helpful assistant that returns travel itineraries based on location, the FCDO guidance from the specified tool, and the weather captured from the displayWeather tool.
        Use the flight information from tool getFlights only to recommend possible flights in the itinerary.
        If there are no flights available generate a sample itinerary and advise them to contact a travel agent.
        Return an itinerary of sites to see and things to do based on the weather.
        Reuse and adapt the prior history if one exists in your memory.
        If the FCDO tool warns against travel DO NOT generate an itinerary.`;

    const result = streamText({
      model: ollama("qwen3:8b"),
      system: prompt,
      messages: allMessages,
      stopWhen: stepCountIs(2),
      tools,
      experimental_telemetry: { isEnabled: true },
      onFinish: async ({ text }) => {
        const guardrailResult = await guards.detect(text);
        console.log(`Guardrail results: ${guardrailResult}`);
      },
    });

    // Return data stream to allow the useChat hook to handle the results as they are streamed through for a better user experience
    return result.toUIMessageStreamResponse();
  } catch (e) {
    console.error(e);
    return new NextResponse(
      "Unable to generate a plan. Please try again later!"
    );
  }
}

```

In our example it will then flag potential requests for personal information:

 ![Screenshot 2025-11-13 at 18.43.44](https://us1.discourse-cdn.com/elastic/original/3X/0/c/0ce1979557031789236f49f7c3c03f9d284bd4b4.jpeg)

## Conclusion

For those building and maintaining AI applications this festive season, telemetry data is essential. Hopefully this simple guide on how to use OpenLit and Elastic to generate OpenTelemetry signals will help you have a more restful on-call this season. The full code is available [here](https://github.com/carlyrichmond/observing-ai-agents).

_Happy holidays!_