APM Java Agent

Jurilz · December 14, 2020, 12:18pm

Kibana version: 7.10.0

Elasticsearch version: 7.10.0

APM Server version: 7.10.1

APM Agent language and version: Java - apm-agent-attach-1.19.0-standalone.jar

Browser version: Firefox 83.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version: docker

Good day,

I'm trying to set up APM (Server and Agent) for my elastic stack, which includes elasticsearch, logstash and kibana. My understanding is that these are all java applications, so I would like to monitore their performance with the apm agent (+ server).

So first I set up the apm server according to these instructions:

my apm-server.yml:

apm-server:
  host: "0.0.0.0:8200"

output.elasticsearch:
  hosts: ["https://es01:9200"]
  username: "elastic"
  password: <my_pw>
  ssl.certificate_authorities: ["/usr/share/elasticsearch/config/certificates/ca/ca.crt"]

The apm-server seems to start successfully, according to the logs:

2020-12-14T10:56:21.039Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(elasticsearch(https://es01:9200))
2020-12-14T10:56:21.040Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2020-12-14T10:56:21.040Z	INFO	[publisher]	pipeline/retry.go:223	  done
2020-12-14T10:56:21.090Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.10.0
2020-12-14T10:56:21.191Z	INFO	[license]	licenser/check.go:35	License is active for Basic
2020-12-14T10:56:21.191Z	INFO	[license]	licenser/check.go:22	Trial license active
2020-12-14T10:56:21.192Z	INFO	[license]	licenser/es_callback.go:51	Elasticsearch license: Trial
2020-12-14T10:56:21.198Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm
2020-12-14T10:56:21.203Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_user_agent
2020-12-14T10:56:21.207Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_user_geo
2020-12-14T10:56:21.210Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_ingest_timestamp
2020-12-14T10:56:21.214Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_remove_span_metadata
2020-12-14T10:56:21.214Z	INFO	[pipelines]	pipeline/register.go:56	Registered Ingest Pipelines successfully.
2020-12-14T10:56:21.216Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.10.0
2020-12-14T10:56:21.635Z	INFO	[index-management]	idxmgmt/manager.go:84	Overwrite ILM setup is disabled. 
2020-12-14T10:56:21.635Z	INFO	[index-management]	idxmgmt/manager.go:203	Set setup.template.name to 'apm-%{[observer.version]}'.
2020-12-14T10:56:21.635Z	INFO	[index-management]	idxmgmt/manager.go:205	Set setup.template.pattern to 'apm-%{[observer.version]}*'.
2020-12-14T10:56:21.665Z	INFO	template/load.go:97	Template apm-7.10.1 already exists and will not be overwritten.
2020-12-14T10:56:21.665Z	INFO	[index-management]	idxmgmt/manager.go:211	Finished loading index template.
2020-12-14T10:56:21.670Z	INFO	[index-management.ilm]	ilm/std.go:139	do not generate ilm policy: exists=true, overwrite=false
2020-12-14T10:56:21.670Z	INFO	[index-management]	idxmgmt/manager.go:242	ILM policy apm-rollover-30-days successfully loaded.
2020-12-14T10:56:21.695Z	INFO	template/load.go:97	Template apm-7.10.1-span already exists and will not be overwritten.
2020-12-14T10:56:21.695Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-span.
2020-12-14T10:56:21.703Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-span successfully generated.
2020-12-14T10:56:21.728Z	INFO	template/load.go:97	Template apm-7.10.1-transaction already exists and will not be overwritten.
2020-12-14T10:56:21.728Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-transaction.
2020-12-14T10:56:21.734Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-transaction successfully generated.
2020-12-14T10:56:21.759Z	INFO	template/load.go:97	Template apm-7.10.1-error already exists and will not be overwritten.
2020-12-14T10:56:21.759Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-error.
2020-12-14T10:56:21.765Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-error successfully generated.
2020-12-14T10:56:21.790Z	INFO	template/load.go:97	Template apm-7.10.1-metric already exists and will not be overwritten.
2020-12-14T10:56:21.791Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-metric.
2020-12-14T10:56:21.801Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-metric successfully generated.
2020-12-14T10:56:21.826Z	INFO	template/load.go:97	Template apm-7.10.1-profile already exists and will not be overwritten.
2020-12-14T10:56:21.827Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-profile.
2020-12-14T10:56:21.833Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-profile successfully generated.
2020-12-14T10:56:21.833Z	INFO	[index-management]	idxmgmt/manager.go:126	Finished index management setup.
2020-12-14T10:56:21.835Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(elasticsearch(https://es01:9200)) established

Next I tried to attach the (Java) apm-agent according to these instructions to all docker containers:

I got the standalone.jar and executed the 'attach.sh.' and got the following output:

+ docker ps --quiet --no-trunc
+ attach
+ docker inspect ed110dcd84294ca42ff1b84fc09e02ebb064243d00f5bdbb4390d57fe6117e8f
+ jq --raw-output .[0].Config.Cmd[0]
+ [[ /usr/local/bin/kibana-docker == java ]]
attach.sh.: 6: [[: not found
+ attach
+ docker inspect cb07cbc9a7c4e876d6d3a4dd6f47f5ed427ba3634a22c751a635cdd5137231a1
+ jq --raw-output .[0].Config.Cmd[0]
+ [[ --strict.perms=false == java ]]
attach.sh.: 6: [[: not found
+ attach
+ docker inspect 40a239ecdcd97aa9437ab2e64615a3e3c7682d37a8636b9f849b216e6e6b707d
+ jq --raw-output .[0].Config.Cmd[0]
+ [[ null == java ]]
attach.sh.: 6: [[: not found
+ attach
+ docker inspect 317ba11ffd29467965d72dc99016b43ce51867631994629e03ec1a0ee7b6451c
+ jq --raw-output .[0].Config.Cmd[0]
+ [[ eswrapper == java ]]
attach.sh.: 6: [[: not found
+ attach
+ docker inspect 82ecc8541003875d25589d62098738c05a4cea6524931b5fb03aaaca0f459479
+ jq --raw-output .[0].Config.Cmd[0]
+ [[ eswrapper == java ]]
attach.sh.: 6: [[: not found
+ attach
+ + docker inspect 443494eff332eacc3d13994b021cb5bba34405615472d65e571c981c21975749
jq --raw-output .[0].Config.Cmd[0]
+ [[ eswrapper == java ]]
attach.sh.: 6: [[: not found
+ attach
+ docker inspect d3b979c4e9b36a2ccc7b24dab7dc39fa9af360096c7e186977bfd57f5b7bbc7e
+ jq --raw-output .[0].Config.Cmd[0]
+ [[ null == java ]]
attach.sh.: 6: [[: not found
+ docker events --filter event=start --format {{.ID}}
+ IFS= read -r container_id

Next I looked at the script and tried to reproduce it myself. So I copied the the standalone.jar into the logstash container and started the agent from within the container with the command (after installing open-jdk-8 in the container):

java -jar /apm-agent-attach-standalone.jar --config server_urls=http://apm-server:8200

From wich got the following message:

2020-12-14 12:11:03.150  INFO Attaching the Elastic APM agent to 1 org.logstash.Logstash -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -Dls.cgroup.cpuacct.path.override=/ -Dls.cgroup.cpu.path.override=/ with arguments {server_urls=http://apm-server:8200}
2020-12-14 12:11:03.662  INFO Done

But kibana /app/home#/tutorial_directory tells me: No data has been received from agents yet

And the apm-server logs don't show any new messages, telling that that an Agent has connected.

Btw: curl "http://apm-server:8200" tells that the apm-server is reachable:

 {
  "build_date": "2020-12-04T22:07:34Z",
  "build_sha": "b7c209e80c4674603447458e62963ed5246b5297",
  "version": "7.10.1"
}

I'm sure, that I'm doing doing something terribly wrong. So any helf would be much appreciated.

Sylvain_Juge · December 14, 2020, 3:42pm

Hi @Jurilz

First, make sure that you have read the troubleshooting section of our documentation.

From what you provided, it seems that you can properly attach to running JVMs using the standalone jar. The next thing to check here is to check log files (or standard output) from those running JVMs.

Also, please note that APM agent does not provide (yet) any automatic instrumentation plugin for Elasticsearch, however, we have plans to do that in the future. See https://github.com/elastic/apm-agent-java/issues/1561 issue that should add meaningful metrics/tracing information.

apm-server is written in Go, there is a separate agent for Go.

Jurilz · December 15, 2020, 11:51am

Thank you @Sylvain_Juge for your support on this topic.

The purpose of this operation was to gather some experience with the APM agent and server. Therefore I tried to attach the agent first to all containers and afterwards particulary to the logstash container. Especially because locally on my computer no other (Java) program is running. But thinking further about the approach, I realized that this is not my current use case.

Therefore I'm setting up some Java Web Servers (tomcat and apache) and will try to monitore their performance with the APM agent and server.

system · January 5, 2021, 7:51am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.