1. Is the Datadog APM Java Agent officially supported for use with Elasticsearch 9.x?
2. What are the specifuc Entitlements(security permissions) required for the dd-java-agent.jar to run as a Java Agent on an Elasticsearch 9.x node?
I have attached a stack trace/screenshot showing the exact error encountered during the Elasticsearch startup, which you may find helpful for your investigation.
The error explicitly states: ENTITLEMENT [outbound_network] not granted.
This confirms that the Agent is being blocked from making necessary network connections due to the new security model.
Perhaps read this as @DavidTurner recently answered
I would assume the same would go for the DD Agent
Specifically from the Docs Referenced
Warning
Don’t use third-party Java agents that attach to the JVM. Such agents can be harmful to Elasticsearch stability and performance. In some cases they may cause nodes to freeze, crash, or fail to start up, or to lose or corrupt your data.
Hi @stephenb ,Thank you for your previous response.
Based on @DavidTurner's earlier answer, my understanding is that any third-party Java agent (including the Datadog agent) is not permitted to attach to the Elasticsearch JVM, as this could negatively impact the stability and performance of Elasticsearch. Is my understanding correct?
If I wanted to have the Datadog agent as one of the plugins trusted by Elasticsearch, could it then be attached to the Elasticsearch JVM to monitor Elasticsearch? If I can not attach it to the Elasticsearch JVM, what else can I do?
Elasticsearch itself exposes an extensive collection of stats and metrics (e.g. via Elastic Agent) so that’s what we’d recommend. General-purpose monitoring tools like Datadog’s agent don’t collect and interpret the Elasticsearch-specific metrics, making it impossible to properly interpret the metrics they do collect.
Thank you for providing this important feedback regarding metric collection and the use of Elastic Agent.
I fully understand and agree with your recommendation that Elastic Agent is the best solution for comprehensive metrics and operational monitoring of the Elasticsearch cluster itself. We will ensure we prioritize the use of native Elastic tools for that purpose.
For clarification, our original goal with the Datadog Java Agent was not for general metrics collection, but specifically for Distributed Tracing (APM). This was to trace API calls from our application services through the Elasticsearch node to connect them within our wider APM environment.
Update: We have received confirmation from Datadog that their Java Agent is not yet officially certified for use within the Elasticsearch 9.x Entitlements sandbox. Therefore, we have stopped our efforts to load the javaagent on the Elasticsearch nodes.
We will proceed with the alternative solution of instrumenting the client applications only to ensure we capture the necessary trace correlation without interfering with Elasticsearch's internal security model.
Thank you for clarifying the best practice for Elasticsearch monitoring.
Right but it’s the same issue here as with metrics: Elasticsearch uses its own very custom (and constantly-changing) framework for handling API calls so I wouldn’t expect a general-purpose APM agent like Datadog to yield any useful information.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
