Kibana version: 7.10.0
Elasticsearch version: 7.10.0
APM Server version: 7.10.1
APM Agent language and version: Java - apm-agent-attach-1.19.0-standalone.jar
Browser version: Firefox 83.0
Original install method (e.g. download page, yum, deb, from source, etc.) and version: docker
Good day,
I'm trying to set up APM (Server and Agent) for my elastic stack, which includes elasticsearch, logstash and kibana. My understanding is that these are all java applications, so I would like to monitore their performance with the apm agent (+ server).
So first I set up the apm server according to these instructions:
my apm-server.yml:
apm-server:
host: "0.0.0.0:8200"
output.elasticsearch:
hosts: ["https://es01:9200"]
username: "elastic"
password: <my_pw>
ssl.certificate_authorities: ["/usr/share/elasticsearch/config/certificates/ca/ca.crt"]
The apm-server seems to start successfully, according to the logs:
2020-12-14T10:56:21.039Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(https://es01:9200))
2020-12-14T10:56:21.040Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2020-12-14T10:56:21.040Z INFO [publisher] pipeline/retry.go:223 done
2020-12-14T10:56:21.090Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
2020-12-14T10:56:21.191Z INFO [license] licenser/check.go:35 License is active for Basic
2020-12-14T10:56:21.191Z INFO [license] licenser/check.go:22 Trial license active
2020-12-14T10:56:21.192Z INFO [license] licenser/es_callback.go:51 Elasticsearch license: Trial
2020-12-14T10:56:21.198Z INFO [pipelines] pipeline/register.go:53 Pipeline already registered: apm
2020-12-14T10:56:21.203Z INFO [pipelines] pipeline/register.go:53 Pipeline already registered: apm_user_agent
2020-12-14T10:56:21.207Z INFO [pipelines] pipeline/register.go:53 Pipeline already registered: apm_user_geo
2020-12-14T10:56:21.210Z INFO [pipelines] pipeline/register.go:53 Pipeline already registered: apm_ingest_timestamp
2020-12-14T10:56:21.214Z INFO [pipelines] pipeline/register.go:53 Pipeline already registered: apm_remove_span_metadata
2020-12-14T10:56:21.214Z INFO [pipelines] pipeline/register.go:56 Registered Ingest Pipelines successfully.
2020-12-14T10:56:21.216Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.0
2020-12-14T10:56:21.635Z INFO [index-management] idxmgmt/manager.go:84 Overwrite ILM setup is disabled.
2020-12-14T10:56:21.635Z INFO [index-management] idxmgmt/manager.go:203 Set setup.template.name to 'apm-%{[observer.version]}'.
2020-12-14T10:56:21.635Z INFO [index-management] idxmgmt/manager.go:205 Set setup.template.pattern to 'apm-%{[observer.version]}*'.
2020-12-14T10:56:21.665Z INFO template/load.go:97 Template apm-7.10.1 already exists and will not be overwritten.
2020-12-14T10:56:21.665Z INFO [index-management] idxmgmt/manager.go:211 Finished loading index template.
2020-12-14T10:56:21.670Z INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2020-12-14T10:56:21.670Z INFO [index-management] idxmgmt/manager.go:242 ILM policy apm-rollover-30-days successfully loaded.
2020-12-14T10:56:21.695Z INFO template/load.go:97 Template apm-7.10.1-span already exists and will not be overwritten.
2020-12-14T10:56:21.695Z INFO [index-management] idxmgmt/manager.go:223 Finished template setup for apm-7.10.1-span.
2020-12-14T10:56:21.703Z INFO [index-management] idxmgmt/manager.go:259 Write alias apm-7.10.1-span successfully generated.
2020-12-14T10:56:21.728Z INFO template/load.go:97 Template apm-7.10.1-transaction already exists and will not be overwritten.
2020-12-14T10:56:21.728Z INFO [index-management] idxmgmt/manager.go:223 Finished template setup for apm-7.10.1-transaction.
2020-12-14T10:56:21.734Z INFO [index-management] idxmgmt/manager.go:259 Write alias apm-7.10.1-transaction successfully generated.
2020-12-14T10:56:21.759Z INFO template/load.go:97 Template apm-7.10.1-error already exists and will not be overwritten.
2020-12-14T10:56:21.759Z INFO [index-management] idxmgmt/manager.go:223 Finished template setup for apm-7.10.1-error.
2020-12-14T10:56:21.765Z INFO [index-management] idxmgmt/manager.go:259 Write alias apm-7.10.1-error successfully generated.
2020-12-14T10:56:21.790Z INFO template/load.go:97 Template apm-7.10.1-metric already exists and will not be overwritten.
2020-12-14T10:56:21.791Z INFO [index-management] idxmgmt/manager.go:223 Finished template setup for apm-7.10.1-metric.
2020-12-14T10:56:21.801Z INFO [index-management] idxmgmt/manager.go:259 Write alias apm-7.10.1-metric successfully generated.
2020-12-14T10:56:21.826Z INFO template/load.go:97 Template apm-7.10.1-profile already exists and will not be overwritten.
2020-12-14T10:56:21.827Z INFO [index-management] idxmgmt/manager.go:223 Finished template setup for apm-7.10.1-profile.
2020-12-14T10:56:21.833Z INFO [index-management] idxmgmt/manager.go:259 Write alias apm-7.10.1-profile successfully generated.
2020-12-14T10:56:21.833Z INFO [index-management] idxmgmt/manager.go:126 Finished index management setup.
2020-12-14T10:56:21.835Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(https://es01:9200)) established
Next I tried to attach the (Java) apm-agent according to these instructions to all docker containers:
I got the standalone.jar and executed the 'attach.sh.' and got the following output:
+ docker ps --quiet --no-trunc
+ attach
+ docker inspect ed110dcd84294ca42ff1b84fc09e02ebb064243d00f5bdbb4390d57fe6117e8f
+ jq --raw-output .[0].Config.Cmd[0]
+ [[ /usr/local/bin/kibana-docker == java ]]
attach.sh.: 6: [[: not found
+ attach
+ docker inspect cb07cbc9a7c4e876d6d3a4dd6f47f5ed427ba3634a22c751a635cdd5137231a1
+ jq --raw-output .[0].Config.Cmd[0]
+ [[ --strict.perms=false == java ]]
attach.sh.: 6: [[: not found
+ attach
+ docker inspect 40a239ecdcd97aa9437ab2e64615a3e3c7682d37a8636b9f849b216e6e6b707d
+ jq --raw-output .[0].Config.Cmd[0]
+ [[ null == java ]]
attach.sh.: 6: [[: not found
+ attach
+ docker inspect 317ba11ffd29467965d72dc99016b43ce51867631994629e03ec1a0ee7b6451c
+ jq --raw-output .[0].Config.Cmd[0]
+ [[ eswrapper == java ]]
attach.sh.: 6: [[: not found
+ attach
+ docker inspect 82ecc8541003875d25589d62098738c05a4cea6524931b5fb03aaaca0f459479
+ jq --raw-output .[0].Config.Cmd[0]
+ [[ eswrapper == java ]]
attach.sh.: 6: [[: not found
+ attach
+ + docker inspect 443494eff332eacc3d13994b021cb5bba34405615472d65e571c981c21975749
jq --raw-output .[0].Config.Cmd[0]
+ [[ eswrapper == java ]]
attach.sh.: 6: [[: not found
+ attach
+ docker inspect d3b979c4e9b36a2ccc7b24dab7dc39fa9af360096c7e186977bfd57f5b7bbc7e
+ jq --raw-output .[0].Config.Cmd[0]
+ [[ null == java ]]
attach.sh.: 6: [[: not found
+ docker events --filter event=start --format {{.ID}}
+ IFS= read -r container_id
Next I looked at the script and tried to reproduce it myself. So I copied the the standalone.jar into the logstash container and started the agent from within the container with the command (after installing open-jdk-8 in the container):
java -jar /apm-agent-attach-standalone.jar --config server_urls=http://apm-server:8200
From wich got the following message:
2020-12-14 12:11:03.150 INFO Attaching the Elastic APM agent to 1 org.logstash.Logstash -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -Dls.cgroup.cpuacct.path.override=/ -Dls.cgroup.cpu.path.override=/ with arguments {server_urls=http://apm-server:8200}
2020-12-14 12:11:03.662 INFO Done
But kibana /app/home#/tutorial_directory tells me: No data has been received from agents yet
And the apm-server logs don't show any new messages, telling that that an Agent has connected.
Btw: curl "http://apm-server:8200" tells that the apm-server is reachable:
{
"build_date": "2020-12-04T22:07:34Z",
"build_sha": "b7c209e80c4674603447458e62963ed5246b5297",
"version": "7.10.1"
}
I'm sure, that I'm doing doing something terribly wrong. So any helf would be much appreciated.