Java Agent not visible in APM Server

Elastic Observability APM
1

Hi Guys
I try to bring the Java Agent up and running but it won't work. In Kibana (APM tab) no agent is displayed.

  • "You have correctly setup APM Server"

  • "Data successfully received from one or more agents"

  • PING from APM server to Wildfly works.

  • Firewall has been configured APM server-side (listen to port 8200)

  • Starting the agent in the Wildfly application log looks positive

2018-11-19 13:50:11.566 [main] INFO co.elastic.apm.configuration.StartupInfo - Starting Elastic APM 1.0.0 as ger-apm-agent on Java 1.8.0_181 (Oracle Corporation) Windows 10 10.0

2018-11-19 13:50:11.570 [main] WARN co.elastic.apm.configuration.StartupInfo - To enable all features and to increase startup times, please configure application_packages

2018-11-19 13:50:11.625 [apm-server-healthcheck] INFO co.elastic.apm.report.ApmServerHealthChecker - Elastic APM server is available: {"ok":{"build_date":"2018-11-09T17:48:30Z","build_sha":"2c46a2919dbe06767caeacbffd0055aab1493991","version":"6.5.0"}}

13:50:12,666 INFO [org.jboss.modules] (main) JBoss Modules version 1.7.0.Final
13:50:14,487 INFO [org.jboss.msc] (main) JBoss MSC version 1.3.2.Final
13:50:14,505 INFO [org.jboss.threads] (main) JBoss Threads version 2.3.1.Final
13:50:14,643 INFO [org.jboss.as] (MSC service thread 1-2) WFLYSRV0049: WildFly Full 12.0.0.Final (WildFly Core 4.0.0.Final) starting

Does anybody of you have a clue how to troubleshoot further?

2

Hi and thanks for trying Elastic APM!

The agent only shows up if the agent has already recorded at least one request in the specified time frame. So make sure to execute some requests on your application and refresh the Kibana page.

Also check that you are using one of the supported web frameworks and that there are no warnings or errors in the log after executing some requests.

Cheers,
Felix

3

Thanks Felix for your prompt reply.

I checked your recommendations: The Wildfly version is officially supported.

Below a debug level trace:

2018-11-19 17:45:03.850 [apm-server-healthcheck] INFO co.elastic.apm.report.ApmServerHealthChecker - Elastic APM server is available: {"ok":{"build_date":"2018-11-09T17:48:30Z","build_sha":"2c46a2919dbe06767caeacbffd0055aab1493991","version":"6.5.0"}}
2018-11-19 17:45:03.886 [main] INFO co.elastic.apm.configuration.StartupInfo - Starting Elastic APM 1.0.0 as nov-apm-agent on Java 11.0.1 (Oracle Corporation) Windows 10 10.0

17:45:40,868 DEBUG [activemq-expiry-reaper-thread] [] [QueueImpl] Cannot expire from jms.queue.ExpiryQueue into jms.queue.ExpiryQueue
17:45:44,241 INFO  [apm-reporter] [] [stdout] 2018-11-19 17:45:44.239 [apm-reporter] INFO co.elastic.apm.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (▒10%)
17:45:44,241 INFO  [apm-reporter] [] [stdout] 2018-11-19 17:45:44.241 [apm-reporter] WARN co.elastic.apm.report.IntakeV2ReportingEventHandler - Read timed out
17:45:44,245 INFO  [apm-reporter] [] [stdout] 2018-11-19 17:45:44.245 [apm-reporter] WARN co.elastic.apm.report.IntakeV2ReportingEventHandler - null
17:45:59,251 INFO  [apm-reporter] [] [stdout] 2018-11-19 17:45:59.251 [apm-reporter] INFO co.elastic.apm.report.IntakeV2ReportingEventHandler - Backing off for 1 seconds (▒10%)
17:45:59,251 INFO  [apm-reporter] [] [stdout] 2018-11-19 17:45:59.251 [apm-reporter] WARN co.elastic.apm.report.IntakeV2ReportingEventHandler - Read timed out
17:45:59,261 INFO  [apm-reporter] [] [stdout] 2018-11-19 17:45:59.261 [apm-reporter] WARN co.elastic.apm.report.IntakeV2ReportingEventHandler - null

How can I fix these warnings?

Thanks
Manuel

4

Indicates that the APM server took too long to handle the request. Is your APM Server under load? The default server_timeout of the agent is 5s. You could try to increase this number to see if it helps.

Having a look at the APM Server logs can hopefully shed some light on why it responds slow.

5

Okay, I got it to work. Thanks! Nevertheless a couple of clues and questions:

Do I understand correctly:

  1. the APM agent sends an onboard message to the APM server
  2. APM servers writes a log entry but at this point the agent is not yet registered in Kibana APM
  3. Connection between agent and server will be closed. (?)
  4. APM server pulls data every 30sec (default)
  5. In case logs are available the agent gets registered in Kibana and shows stats/charts/etc.

Question/Clue:
a) Why wont the agent be registered in the onboard message?
b) In my scenario, a firewall blocked the pull from APM server. It is required that the APM server can establish a connection to the agent; not only from agent to the server.

Thanks
Manuel

6

At startup, the agent does a health check to the server. The server logs the access to /healthcheck but does not write anything into Elasticsearch yet.

However, when the APM server starts, it does write an onboarding document to Elasticsearch and the "You have correctly setup APM Server" check just verifies if the onboarding document of the APM Server is present.

The server does not pull anything from the agent. Only the agent sends data to the Server, not vice versa.

7

Simulate the agent by trying to send a properly formatted metric to the APM server. Refer to the intake format below:

https://www.elastic.co/guide/en/apm/server/current/example-intakev2-events.html

also see: https://www.elastic.co/guide/en/apm/server/current/intake-api.html

I used curl against the APM server to see if it was accepting metrics and passing it on to Elasticsearch. That way, I can rule out connection issues between Elastic APM and Elasticsearch.

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.