Elastic Agent to OpenTelemetry Collector

[chadr]

Hi all,

I am researching the capabilities of the current and future roadmap for the Elastic Agent. We have a very diverse infra/app/coding language environment.

Specifically, I looking to understand if the Elastic Agent can or will be able to send Logs, Metrics, Traces directly to an OpenTelemetry collector. While we are still planning on sending data to the ELK stack, we have a lot of data (>1TB/minute) and do a ton of preprocessing in our OpenTelemetry collectors before writing to the multiple backend systems.

I found some great information here in an older post.

Elastic APM agent VS OpenTelemetry client

Now that Elastic Stack has native support for OpenTelemetry I guess there are (at least) two ways to do APM from a .Net application (the same question is probably valid for java and others as well). Using the Elastic APM .NET Agent. Using the The OpenTelemetry .NET Client. Looking at Github stats (number of stars, contributers, pulse) OpenTelemetry is way ahead of the Elastic one. Also OT is an open standard implemented by several vendors. As an Elastic Stack customer, is there any reason to…

The post suggests that the Elastic Agent supports OTEL but only by using the OTEL SDK.

I have also found some interesting changes in the upcoming Elastic Agent, in particular, the new Elastic Agent Shipper which appears to allow the creation of custom shippers.

GitHub - elastic/elastic-agent-shipper: Data shipper for the Elastic Agent -...

I don't see anything that would support OpenTelemetry specifically per our use case.

Does anyone have thoughts on how we could support an enterprise Elastic Agent to OpenTelemetry Collector without the need of the SDK?

[GregKalapos]

Hi @chadr,

just to make sure we are on the same page, I'd like to clarify that we have 2 things here, which have very similar names, but they are completely different components:

• Elastic agent is an external process which is a unified way to add monitoring for logs, metrics, and other types of data to a host. So what it offers you is a unified and easy to use way to start shipping e.g. new logs, but in general the elastic agent collects these logs out of process.
• Elastic APM Agents target specific technologies (so we have multiple Elastic APM Agents, each target a specific supported technology) and they always run in-process. So you inject the specific Elastic APM Agent (e.g. the Java agent) into a specific process (e.g. a Java process) and it starts monitoring your process. This gives significantly deeper visibility into an application - so you not only collect and ship logs, but the agent also monitors the internal mechanisms of the given process and can e.g. collect exceptions (even if they are not logged) or data related to specific database systems (like the db query itself), etc.

Now, one note here: I understand this is very confusing and I definitely don't blame anyone who is not aware of the difference. This is not the best naming within Elastic.

The discuss entry you linked talks about Elastic APM Agents (in that case specifically about the .NET Agent) and as you correctly say, the post basically states that Elastic APM Agents support the OTel API.

However, the GitHub repository you linked is about the Elastic Agent - that's not APM related. So that is about a different component.

So, my question to your question

• Which component do you actually mean? Do you want the Elastic Agent to ship via OTLP to the OTel collector, or do you want an Elastic APM Agent to ship data via OTLP to the OTel collector?

With that, finally I'll also try to answer your question: currently the answer is actually the same for both: we continuously look into broadening our OTel support and you can expect to see more. However, in both cases, we don't have any specific decision or timeline to share, and we don't even have a GitHub issue to linked to, so even if we'd offer this, that won't happen in the near future.

I'd be happy to hear which component you are interested in (Elastic Agent vs. Elastic APM Agents). I personally work on Elastic APM Agents and we discuss potentially adding OTLP support to our existing APM Agents (nothing decided here, just discussions - and this comment is only for Elastic APM Agents and not for the Elastic Agent) - there are good arguments for and against it, and user feedback is very appreciated on this subject.

Currently this post is tagged with APM, if this is about the Elastic Agent (and not Elastic APM Agents), we can change the tag and people from the Elastic Agent team can chime in.

[chadr]

Thanks @GregKalapos - Specifically, we are targeting using the Elastic Agent as a base for our ability to automatically send Logs/Metrics/Traces directly to our Observability platform which uses OTEL collectors to pass on to our ELK Backend. Currently we have just about every vended and open source solutions in our 1M+ servers running in multiple on-premise and cloud data centers.

As part of our enterprise strategy we are looking to take advantage of a nirvana solution to use a single agent for capturing and writing all LMT to our OTEL collector/processors and distribute to our backend storage.

It looks a bit like this:

Agent -> Collector -> Processor -> Storage -> Presentation

Where the Agent (could be) Elastic Agent writing to OTEL and our OTEL writing to ELK.

Today we have at least 12+ Agents and 5+ Storage solutions.

Does that help?

[GregKalapos]

Thanks for clarifying @chadr, I see. So in this we are talking about the Elastic Agent. I moved this entry to the Elastic Agent tag.

Since I focus on APM, and not the Elastic Agent, I can't really add more to this. I think the answer is still the same, nevertheless, let's see if colleagues from that team have anything more to share.