Context
Our Elastic journey has been interesting and filled with trial and error. Around every corner we discover a different (and often better) way of doing the same thing, but it usually means redoing large parts of our setup. We’re getting a bit tired of rebuilding everything, so I’m hoping to get some advice that helps us skip a few corners and rediscoveries.
Below I have included our observability journey, and the questions are at the bottom.
Our Journey
ELK
We wanted to introduce observability for our applications and already had some prior experience with the ELK stack. We already used Elasticsearch, but added Kibana and started to push logs via Logstash.
We added a custom log line that we sent to a separate index, which we used for our dashboards and statistics.
2026-01-01 12:00:00 thread-01 MyClass.java: LOGSTASH [status:200][ip:xx.xx.xx.xx][duration:12]
Agents, Integration, Custom Logs (Filestream) & Pipeline
After running into some issues with Logstash, we discovered that Fleet Agents could install the Custom Logs (Filestream) integration and use our own ingest pipelines, effectively replacing Logstash entirely. At the time this felt like a better and more modern solution.
Kibana Observability
Later, we discovered Kibana Observability and realized that our existing solution didn’t integrate well with it. We then installed the Elastic APM integration on our agents and started using the Elastic Java Agent.
OpenTelemetry standard
After seeing how many other tools and frameworks we use support OpenTelemetry out of the box, we wanted to move toward the OTEL standard. However, we couldn’t get the standard OTEL SDKs to work directly with the Elastic APM integration or Kibana's Observability solution, so we switched from the Elastic Java Agent to the Elastic OTEL Java Agent.
This is where we started noticing feature gaps.
When using the Elastic OTEL Java Agent together with the Elastic APM integration:
- Metric attributes appear to be dropped
- Unmapped attributes are rewritten as
labels.xxxx - Multiple Kibana's Observability Metrics visualizations not working
- Other expected OTEL behaviors don’t seem to be fully supported
For example:
// pseudo
meter.histogramBuilder("my.histogram.name")
.setUnit("ms")
.ofLongs()
.build();
histrogram.record(12, Attributes.of("some.attribute", "some.value"));
The recorded metric ends up looking like this, with the attribute missing:
"my.histogram.name": { "counts": [4], "values": [6250] }
The Future (Maybe?)
After more research, it seems that we should not be using the Elastic APM integration at all, but instead:
- Use the OpenTelemetry integration / EDOT OTEL Collector
- Install the System OpenTelemetry Assets integration
- Use default and standard OTEL SDKs and tooling
- Let Elastic/Kibana consume OTEL semantic conventions over ECS
If that understanding is correct, this would allow us to stick to standard OTEL tooling while still using Elastic Observability effectively.
My lingering questions
- Will using the OpenTelemetry integration / EDOT OTEL Collector allow us to use standard OTEL tools without
labels.xxx, retain all metric attributes, and still have Kibana Observability fully working? - If we move to the OpenTelemetry integration / EDOT OTEL Collector, are we finally on the “recommended” path, or are we likely to discover yet another better approach and end up rebuilding again?
- Is it currently possible to use default OpenTelemetry tools end-to-end together with the Elastic Observability solution?
