ELK APM

[rj23495 (rajat jain)]

I installed APM agent as jar file for tomcat machine and was able to get the metrics but i cannot see any graphs in requests or transactions per minute. PFA screenshot. The time range is last 7 days.

[simitt (Silvia Mitter)]

Hi @rj23495,
can you please check your agent and APM Server logs for any errors?

[rj23495 (rajat jain)]

There are no errors in logs, i checked it.
2020-04-03T06:44:39.105Z INFO instance/beat.go:611 Home path: [/usr/share/apm-server] Config path: [/usr/share/apm-server] Data path: [/usr/share/apm-server/data] Logs path: [/usr/share/apm-server/logs] 2020-04-03T06:44:39.106Z INFO instance/beat.go:618 Beat UUID: 3d469aea-72a2-40cb-9b00-380cd72bef63 2020-04-03T06:44:39.106Z INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed 2020-04-03T06:44:39.106Z INFO [beat] instance/beat.go:931 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/apm-server", "data": "/usr/share/apm-server/data", "home": "/usr/share/apm-server", "logs": "/usr/share/apm-server/logs"}, "type": "apm-server", "uuid": "3d469aea-72a2-40cb-9b00-380cd72bef63"}}} 2020-04-03T06:44:39.106Z INFO [beat] instance/beat.go:940 Build info {"system_info": {"build": {"commit": "dd4a857d44e17539e5c0bd0db3679b6c091f53ba", "libbeat": "6.7.2", "time": "2019-04-29T08:01:17.000Z", "version": "6.7.2"}}} 2020-04-03T06:44:39.107Z INFO [beat] instance/beat.go:943 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":16,"version":"go1.10.8"}}} 2020-04-03T06:44:39.108Z INFO [beat] instance/beat.go:947 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-03-11T05:30:33Z","containerized":true,"name":"apm-server-5ddf64b7c4-bfvk6","ip":["127.0.0.1/8","172.16.163.17/25"],"kernel_version":"3.10.0-957.21.3.el7.x86_64","mac":["c2:94:0e:4e:e6:5f"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}} 2020-04-03T06:44:39.108Z INFO [beat] instance/beat.go:976 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/apm-server", "exe": "/usr/share/apm-server/apm-server", "name": "apm-server", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2020-04-03T06:44:38.500Z"}}} 2020-04-03T06:44:39.108Z INFO instance/beat.go:280 Setup Beat: apm-server; Version: 6.7.2 2020-04-03T06:44:39.108Z INFO elasticsearch/client.go:164 Elasticsearch url: http://es-sg-yv71j4thx0002toms.elasticsearch.aliyuncs.com:9200 2020-04-03T06:44:39.109Z INFO [publisher] pipeline/module.go:110 Beat name: apm-server-5ddf64b7c4-bfvk6 2020-04-03T06:44:39.109Z INFO [beater] beater/beater.go:110 No pipeline callback registered 2020-04-03T06:44:39.109Z INFO instance/beat.go:402 apm-server start running. 2020-04-03T06:44:39.109Z INFO [beater] beater/beater.go:208 self instrumentation is disabled 2020-04-03T06:44:39.110Z INFO [beater] beater/beater.go:146 host resolved from 0.0.0.0:8200 to [::]:8200 2020-04-03T06:44:39.110Z INFO [handler] beater/common_handlers.go:163 Path /v1/client-side/sourcemaps added to request handler 2020-04-03T06:44:39.110Z INFO [handler] beater/common_handlers.go:163 Path /v1/rum/sourcemaps added to request handler 2020-04-03T06:44:39.110Z INFO [handler] beater/common_handlers.go:163 Path /v1/client-side/transactions added to request handler 2020-04-03T06:44:39.110Z INFO [handler] beater/common_handlers.go:163 Path /v1/errors added to request handler 2020-04-03T06:44:39.110Z INFO [handler] beater/common_handlers.go:163 Path /v1/client-side/errors added to request handler 2020-04-03T06:44:39.110Z INFO [handler] beater/common_handlers.go:163 Path /v1/metrics added to request handler 2020-04-03T06:44:39.110Z INFO [handler] beater/common_handlers.go:163 Path /assets/v1/sourcemaps added to request handler 2020-04-03T06:44:39.110Z INFO [handler] beater/common_handlers.go:163 Path /v1/transactions added to request handler 2020-04-03T06:44:39.110Z INFO [handler] beater/common_handlers.go:163 Path /v1/rum/transactions added to request handler 2020-04-03T06:44:39.110Z INFO [handler] beater/common_handlers.go:163 Path /v1/rum/errors added to request handler 2020-04-03T06:44:39.110Z INFO [handler] beater/common_handlers.go:169 Path /intake/v2/events added to request handler 2020-04-03T06:44:39.110Z INFO [handler] beater/common_handlers.go:169 Path /intake/v2/rum/events added to request handler 2020-04-03T06:44:39.110Z INFO [server] beater/server.go:66 Starting apm-server [dd4a857d44e17539e5c0bd0db3679b6c091f53ba built 2019-04-29 08:01:17 +0000 UTC]. Hit CTRL-C to stop it. 2020-04-03T06:44:39.110Z INFO [server] beater/server.go:67 Listening on: [::]:8200 2020-04-03T06:44:39.110Z INFO [server] beater/server.go:72 RUM endpoints disabled 2020-04-03T06:44:39.111Z INFO [request] beater/common_handlers.go:272 handled request {"request_id": "cf2d5b68-6a50-40a0-b3cf-5eba62ef43ad", "method": "GET", "URL": "/", "content_length": 0, "remote_address": "127.0.0.1", "user-agent": "Go-http-client/1.1", "response_code": 200} 2020-04-03T06:44:39.111Z INFO [http_client] beater/client.go:49 HTTP Server ready 2020-04-03T06:44:39.111Z INFO [onboarding] beater/onboarding.go:36 Publishing onboarding document 2020-04-03T06:44:40.112Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(http://es-sg-yv71j4thx0002toms.elasticsearch.aliyuncs.com:9200)) 2020-04-03T06:44:40.121Z INFO elasticsearch/client.go:739 Attempting to connect to Elasticsearch version 6.7.0 2020-04-03T06:44:40.124Z INFO template/load.go:128 Template already exists and will not be overwritten. 2020-04-03T06:44:40.124Z INFO instance/beat.go:889 Template successfully loaded. 2020-04-03T06:44:40.124Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch(http://es-sg-yv71j4thx0002toms.elasticsearch.aliyuncs.com:9200)) established 2020-04-03T06:44:50.142Z INFO [request] beater/common_handlers.go:272 handled request {"request_id": "12cbbb31-557a-4552-8d2b-dc960d748c2c", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.16.133.64", "user-agent": "elasticapm-java/1.14.0", "response_code": 202}

[rj23495 (rajat jain)]

@simitt And also one question is the queue for ELK APM persistent, i mean if elasticsearch is not ingesting events, will the events be queued in ELK apm queue?