Elastic Observability vs OpenTelemetry: Are We Finally on the "Right" Path?

Elastic Observability
1

Context

Our Elastic journey has been interesting and filled with trial and error. Around every corner we discover a different (and often better) way of doing the same thing, but it usually means redoing large parts of our setup. We’re getting a bit tired of rebuilding everything, so I’m hoping to get some advice that helps us skip a few corners and rediscoveries.

Below I have included our observability journey, and the questions are at the bottom.

Our Journey

ELK

We wanted to introduce observability for our applications and already had some prior experience with the ELK stack. We already used Elasticsearch, but added Kibana and started to push logs via Logstash.

We added a custom log line that we sent to a separate index, which we used for our dashboards and statistics.

2026-01-01 12:00:00 thread-01 MyClass.java: LOGSTASH [status:200][ip:xx.xx.xx.xx][duration:12]

Agents, Integration, Custom Logs (Filestream) & Pipeline

After running into some issues with Logstash, we discovered that Fleet Agents could install the Custom Logs (Filestream) integration and use our own ingest pipelines, effectively replacing Logstash entirely. At the time this felt like a better and more modern solution.

Kibana Observability

Later, we discovered Kibana Observability and realized that our existing solution didn’t integrate well with it. We then installed the Elastic APM integration on our agents and started using the Elastic Java Agent.

OpenTelemetry standard

After seeing how many other tools and frameworks we use support OpenTelemetry out of the box, we wanted to move toward the OTEL standard. However, we couldn’t get the standard OTEL SDKs to work directly with the Elastic APM integration or Kibana's Observability solution, so we switched from the Elastic Java Agent to the Elastic OTEL Java Agent.

This is where we started noticing feature gaps.

When using the Elastic OTEL Java Agent together with the Elastic APM integration:

  • Metric attributes appear to be dropped
  • Unmapped attributes are rewritten as labels.xxxx
  • Multiple Kibana's Observability Metrics visualizations not working
  • Other expected OTEL behaviors don’t seem to be fully supported

For example:

// pseudo
meter.histogramBuilder("my.histogram.name")  
        .setUnit("ms")  
        .ofLongs()  
        .build();
histrogram.record(12, Attributes.of("some.attribute", "some.value"));

The recorded metric ends up looking like this, with the attribute missing:

"my.histogram.name": { "counts": [4], "values": [6250] }

The Future (Maybe?)

After more research, it seems that we should not be using the Elastic APM integration at all, but instead:

  • Use the OpenTelemetry integration / EDOT OTEL Collector
  • Install the System OpenTelemetry Assets integration
  • Use default and standard OTEL SDKs and tooling
  • Let Elastic/Kibana consume OTEL semantic conventions over ECS

If that understanding is correct, this would allow us to stick to standard OTEL tooling while still using Elastic Observability effectively.

My lingering questions

  • Will using the OpenTelemetry integration / EDOT OTEL Collector allow us to use standard OTEL tools without labels.xxx, retain all metric attributes, and still have Kibana Observability fully working?
  • If we move to the OpenTelemetry integration / EDOT OTEL Collector, are we finally on the “recommended” path, or are we likely to discover yet another better approach and end up rebuilding again?
  • Is it currently possible to use default OpenTelemetry tools end-to-end together with the Elastic Observability solution?
3 Likes
2

eager to know more about that

1 Like
3

We have similar concerns. Thanks for this post.

4

Hi Christoffer,

thank you for sharing your observability journey and the questions you have.

When using the EDOT Java agent or OTel vanilla you should ingest the data through an EDOT/OTel collector and/or Elastic Cloud Managed OTLP endpoint (reference architecture). The details depend on your environment. This ingestion path stores the telemetry data in Elasticsearch in the OTel-native format.

  • Is it currently possible to use default OpenTelemetry tools end-to-end together with the Elastic Observability solution?

You can use OTel vanilla SDKs/agents and an OTel collector to collect and ingest telemetry data. If you don't have the Elastic Cloud Managed OTLP endpoint on your ingest path you should tune the OTel contrib collector to have the right processing of data.

  • If we move to the OpenTelemetry integration / EDOT OTEL Collector, are we finally on the “recommended” path, or are we likely to discover yet another better approach and end up rebuilding again?

If you use an OTel-native ingestion path, that stores the telemetry in the OTel-native format in Elasticsearch, you should be on the right path. The APM server should not be part of this ingestion path because this would result in the telemetry data being stored in ECS format.

  • Will using the OpenTelemetry integration / EDOT OTEL Collector allow us to use standard OTEL tools without labels.xxx, retain all metric attributes, and still have Kibana Observability fully working?

If you use an OTel-native ingest path, you will have OTel's span, metrics and resource attributes. We have the Elastic's OTel demo environment available. It runs the OTel demo with a mix of EDOTs and OTel vanilla SDKs/agents. Maybe have a look and explore a bit. I recommend to use Discover to explore trace and metrics documents of the ingested data.

What kind of deployment of the Elastic Stack are you using and what version?

5

Thank you very much for the detailed answer.

We are running an on‑prem installation of Elasticsearch and Kibana. I believe most of our software is version 9.1.5. We are currently working on upgrading past 9.2+ as I believe the EDOT collector will be part of the 9.2+ Fleet Agents by default. Though haven't research if this means we don't need to install a separate collector or not. Ref.: Elastic Agent as an OpenTelemetry Collector

I have a few follow‑up questions to really hammer this home and make sure we’re aligning with the recommended architecture.

If I understand correctly, using an OTel‑native ingestion path (OTel SDKs/agents → OTel/EDOT Collector → Elasticsearch) will allow us to ingest all expected APM data.

What I’m still unsure about is whether Kibana’s Observability features will work out of the box in that scenario.

I believe I’ve seen references to installing specific content / assets packs (for example System OpenTelemetry Assets) to have Kibana correctly support OpenTelemetry semantic conventions, is that still required or recommended or needed?

You didn’t explicitly mention the EDOT Collector, which made me wonder:

  • Can we use the standard OpenTelemetry Collector instead of Elastic’s EDOT Collector?
  • If both are supported, what concrete benefits does the EDOT Collector provide over the vanilla collector?

If we fully move to an OTel Collector based ingestion path, am I correct in assuming that we can drop both:

  • the Elastic APM Agent, and
  • the elastic‑otel‑javaagent

…and instead rely on standard OpenTelemetry tooling such as:

  • native OTEL SDKs (io.opentelemetry:*)
  • framework integrations (for example Quarkus Micrometer OTEL, WildFly Micrometer OTEL bridge, etc.)

In other words: is the Elastic OTEL Java Agent optional if we are already instrumenting via SDKs/frameworks and exporting to a collector?

(Completely understandable if Java‑specific details are out of scope.)

Today we are attaching elastic-otel-javaagent-1.6.0.jar to the JVM and manually enriching traces using io.opentelemetry:opentelemetry-api. From your explanation, I understand that this results in data being converted and stored in ECS format, due to the Elastic APM integration.

If we introduce an OTel Collector and start ingesting OTEL‑native data in parallel during migration:

  • Will this cause issues for Kibana Observability, given that both ECS‑formatted and OTEL‑native telemetry would coexist temporarily?
  • Is there a recommended migration strategy to avoid breaking dashboards, correlations, or UI features while transitioning?
6

I will try to cover your questions in one reply.

  • You didn’t explicitly mention the EDOT Collector, which made me wonder

You are right. I did not mention the EDOT Collector explicitly only because I had the impression you are favouring the OTel contrib collector. The benefit of EDOT is that it contains an opinionated configuration for using OpenTelemetry with Elastic Observability. This applies to the EDOT SDKs and to the EDOT Collector. The EDOT SDKs are the upstream SDKs with an opinionated configuration in terms of default configuration settings and auto-instrumentation configuration (e.g. what instrumentation packages are enabled by default) in case of Java you can find more details here (config) and here (features). The EDOT compared to upstream OpenTelemetry page gives an overview on the differences between EDOT and upstream. The Elastic features available with EDOT table gives an overview on known Elastic APM features and their compatibility with EDOT. Known limitations of EDOT/OTel compared to Elastic APM data collection are documented too. For the EDOT SDKs we maintain a Feature overview table compared to Elastic APM agents. There are a few features like central configuration of EDOT SDKs that is currently only available with EDOTs and not with OTel vanilla SDKs/agents. We're working on contributing features to upstream e.g. the central configuration (issue#2546). If you are not using features only available in EDOT SDKs, you should be fine if you would like to choose OTel vanilla. Just make sure the data is collected and ingested expected by the integrations and pre-build dashboards. The EDOT collector contains several components like receivers, exporters, processors, connectors, extensions and providers (list of components). The benefit is you don't have to build your own custom collector with the necessary components included (see custom collector build) or make sure a contrib collector contains the required components. You can choose what better suites you. Using EDOTs and their benefits or using upstream vanilla/contrib.

  • What I’m still unsure about is whether Kibana’s Observability features will work out of the box in that scenario.

What features are you using in Kibana Observability? There are some limitations because some integrations and pre-build dashboards are designed for ECS-formatted data.

  • I believe I’ve seen references to installing specific content / assets packs (for example System OpenTelemetry Assets) to have Kibana correctly support OpenTelemetry semantic conventions, is that still required or recommended or needed?

We are building content packs for OpenTelemetry. The OpenTelemetry project is working on making things stable but changes are expected that can lead to changes in semantic conventions. Content packs are good to react to such changes in a flexible way. We can update the contained pre-build dashboards to be compatible with the updated semantic conventions.

  • In other words: is the Elastic OTEL Java Agent optional if we are already instrumenting via SDKs/frameworks and exporting to a collector?

IIUC this should be possible if you would like to do so. You point the OTLP exporter of the SDKs/frameworks to the OTLP receiver of your collector. What you should keep in mind is that framework integrations have to comply to OTel semantic conventions too. If they are not compliant with OTel semantic conventions you would have to build custom dashboards to visualise the data because integrations and pre-build dashboards are not finding the expected data they are looking for based on semantic conventions. You also need to check if the SDKs and the framework integrations collect the data you are looking for that the EDOT Java agent may collect out-of-the-box because of the opinionated configuration.

  • Today we are attaching elastic-otel-javaagent-1.6.0.jar to the JVM and manually enriching traces using io.opentelemetry:opentelemetry-api. From your explanation, I understand that this results in data being converted and stored in ECS format, due to the Elastic APM integration.

Right. If you run an EDOT collector you point the EDOT Java agent OTLP exporter to the collector and configure the Elasticsearch exporter to send the data to Elasticsearch:

exporters: elasticsearch/otel: endpoints: [ "${env:ELASTIC_ENDPOINT}" ] api_key: ${env:ELASTIC_API_KEY} mapping: mode: otel

I took this snippet from the sample configuration file for a standalone EDOT collector that is linked in the default configuration docs.

The data is then stored in OTel-native format. Such data is stored in OTel-native data streams.

  • If we introduce an OTel Collector and start ingesting OTEL‑native data in parallel during migration: Will this cause issues for Kibana Observability, given that both ECS‑formatted and OTEL‑native telemetry would coexist temporarily?

The data is stored in different data streams. A comparison between OpenTelemetry data streams to Elastic APM is available here. IIUC your question right, I would not expect issues.

  • Is there a recommended migration strategy to avoid breaking dashboards, correlations, or UI features while transitioning?

Because of the differences in ECS data and OTel-native data you have to update your assets like custom dashboards. This is necessary because the concepts are different. Taking JVM metrics as example. With Elastic APM Java agent we had the jvm.memory.heap.used metric and in OTel we have jvm.memory.used that needs to be filtered and aggregated on the metric attribute jvm.memory.type=heap. Replacing the queries of the dashboard panels with ES|QL queries enables the use of the COALESCE function. This allows to show ECS and OTel-native metrics in the same dashboard panels. In cases where OTel uses histograms, you would have to add a new dashboard panel to the dashboard to visualise it. In Java you would need to set OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta to enable delta temporality for the histograms to send them to Elasticsearch.

I hope I covered your questions. Let me know if I missed something. I avoided to write "it depends" on your environment. Please keep in mind that there can be details that rely do.

1 Like
7

Again, thank you for your detailed response.

To summarize my current understanding, if we want to use standard OpenTelemetry tooling with Elasticsearch and Kibana Observability (assuming the frameworks and tooling follow OpenTelemetry semantic conventions):

  • For on‑prem deployments, it is recommended to use the EDOT Collector over a vanilla OTel collector, because it reduces mandatory and manual configuration.

  • Using standard OTel tooling with the EDOT Collector results in data being stored in OTel‑native format instead of ECS...

  • ... which means we should expect to:

    • Adapt Kibana Observability visualizations and queries where prebuilt assets are ECS‑specific.
    • Install and maintain OpenTelemetry content packs.
    • Configure OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta to support metric histograms in Elasticsearch.

  • ... and will result in:

    • No longer get labels.xxx attributes
    • No longer lose Metric attributes

I have no preference when it comes to EDOT Collector vs. the OTel Collector, my main issues have been:

  • We haven't been able to use third-party-software's OTel support out of the box with Kibana Observability.
  • Losing attributes when using the Java OTel SDK.
  • Not been able to fully utilize Metrics as we have lost attributes.
  • Not been able to use OTel SDK produced histograms with Kibana Visualizations.