APM "setup" shows success but no APM data available

Elastic Observability APM
21 
[root@kafka-lagmonitor-558bdbdf9c-55b8t /]# echo $JAVA_OPTS
-javaagent:elastic-apm-agent-1.1.0.jar -Delastic.apm.service_name=kafka_lagmonitor -Delastic.apm.application_packages=com.cedexis.lagmonitor -Delastic.apm.server_urls=http://10.95.96.144:8200

[root@kafka-lagmonitor-558bdbdf9c-55b8t /]# ps ax | grep java
1 ?        Ssl   19:40 java -Dlogback.configurationFile=./logback.xmli -javaagent:elastic-apm-agent-1.1.0.jar -Delastic.apm.service_name=kafka_lagmonitor -Delastic.apm.application_packages=com.cedexis.lagmonitor -Delastic.apm.server_urls=http://10.95.96.144:8200 -jar /kafka_lagmonitor-all.jar
22

@ethrbunny can you update JAVA_OPTS with adding to it log_level parameter:

-Delastic.apm.log_level=DEBUG

and resend your logs again.

23

@ethrbunny did you try this?

24 
[root@playground kafka_lagmonitor]# kubectl --namespace=kafka logs kafka-lagmonitor-558bdbdf9c-nt99h                 
2019-01-10 11:26:00.869 [apm-server-healthcheck] DEBUG co.elastic.apm.report.ApmServerHealthChecker - Starting healthcheck to http://10.95.96.144:8200/
2019-01-10 11:26:00.920 [main] INFO co.elastic.apm.configuration.StartupInfo - Starting Elastic APM 1.1.0 as kafka_lagmonitor on Java 1.8.0_181 (Oracle Corporation) Linux 3.10.0-862.14.4.el7.x86_64
2019-01-10 11:26:00.920 [main] DEBUG co.elastic.apm.configuration.StartupInfo - service_name: 'kafka_lagmonitor' (source: Java System Properties) 
2019-01-10 11:26:00.921 [main] DEBUG co.elastic.apm.configuration.StartupInfo - server_urls: 'http://10.95.96.144:8200' (source: Java System Properties)
2019-01-10 11:26:00.921 [main] DEBUG co.elastic.apm.configuration.StartupInfo - application_packages: 'com.cedexis.lagmonitor' (source: Java System Properties)
2019-01-10 11:26:00.921 [main] DEBUG co.elastic.apm.configuration.StartupInfo - log_level: 'DEBUG' (source: Java System Properties)
2019-01-10 11:26:00.958 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applying advice co.elastic.apm.plugin.api.ElasticApmApiInstrumentation$StartTransactionInstrumentation

...

followed by this:

2019-01-10 11:26:00.981 [main] DEBUG co.elastic.apm.bci.ElasticApmAgent - Applied 43 advices
2019-01-10 11:26:00.990 [apm-server-healthcheck] INFO co.elastic.apm.report.ApmServerHealthChecker - Elastic APM server is available: {"ok":{"build_date":"2018-12-06T18:49:16Z","build_sha":"c784db0b58d2f70db7582dc80850e08a32ba45e5","version":"6.5.3"}}

So it does appear to find the apm-server...

25

@ethrbunny did you try this?

26

Yes - that's the output above ^^

27

I added trace annotation to the code. Now I can see data in the 'traces' panel but still nothing in the 'services' panel.

Unf "response times" and "requests per minute" charts are both empty. Odd because "response times" charts shows "avg: 18ms" at the top.

"response time distribution" shows some values.

28

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.