How to debug apm agent

Elastic Observability APM
1

Hi,

I have integrated apm agents to my app but the data is not getting populated. All the apm server are behind an elb and the agent is point to elb.

I have verified the app machine is able to connect to elb by running
curl -XGET http://:80

{
"build_date": "2020-03-26T04:23:50Z",
"build_sha": "8827b43d24eaf42391ea815bb9fa12bdef5d6bce",
"version": "7.6.2"
}

I have also verified in apm server logs whether the requests are coming.
It is not.

How do I proceed further

2

You should be able to see in the application startup logs the agent (assuming Java as tagged) you should see the agent trying to connect to the APM server which runs on port 8200. You have an ELB in the middle that you appear to have mapped to port 80. So I would make sure your agent configuration server_urls reflect that.

You can also up the log level on the agent side
https://www.elastic.co/guide/en/apm/agent/java/current/config-logging.html

We will probably need to see some of the logs from the agent side to help diagnose.

With normal logging setting you should see similar lines in the logs, the last one showing the agent is connected.

2020-08-26 08:31:28,898 [main] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2020-08-26 08:31:29,966 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.17.0 as car-value-estimator on Java 1.8.0_172 (Oracle Corporation) Mac OS X 10.15.6
2020-08-26 08:31:29,967 [main] WARN  co.elastic.apm.agent.configuration.StartupInfo - To enable all features and to increase startup times, please configure application_packages
2020-08-26 08:31:29,997 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2020-08-26 08:31:30,143 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2020-08-11T18:53:16Z",  "build_sha": "41f41134f4dfe91966765f5a712c7302e71bc56d",  "version": "7.9.0"}

...

3

I am not able to see the last log lines you have shown during the application is starting up.

I am able to see the log line I printed myself just before ElasticApmAttacher.attach(configs);.

Which is
INFO [2020-08-26 20:07:43,117] ElasticAPMRegistry: Registering apm with following info ElasticAPMConfiguration{enabled=true, environment=prod, serviceName='ss', applicationPackages=[ss], exclusionPackages=null, apmUrl=<actual_ip>:80} BaseConfig{circuitBreakerEnabled=true, enableLogCorrelation=true, transactionSampleRate=0.25}

Does the dropwizard version has anything to do with this?

4

Please take a look at the Supported framework

https://www.elastic.co/guide/en/apm/agent/java/current/supported-technologies-details.html

I don't think I see dropwizard at this time. That does not mean you can not use the agent it just means you may / will need to use the Public API to instrument your app.

All that said it seems that your agent is not connecting with the server.

Please double check how you are setting up the agent.

https://www.elastic.co/guide/en/apm/agent/java/current/setup.html

5

Got the root cause of the issue.

Stephen, Dropwizard is nothing but a container, hence apm is supported there.

The cause was my service was running at java 1.8.0_24 which is not supported
https://www.elastic.co/guide/en/apm/agent/java/current/supported-technologies-details.html#supported-java-versions

An action item for APM team here is i didn't get any error log from APM attacher, indicating this. Please address this.

2 Likes
6

Thanks... good to know

Per the docs on the page you referred to, you should have got a message like this in the logs... Can you check and see if it was there?

Failed to start agent - JVM version  not supported: 1.8.0_31
7

Nope I didnt get the error message. Thats why I wrote can APM developers look into this why error log was not getting generated in 1.8.0_24

8

I have passed this along thanks for the information.

9

That might be because you have looked at the wrong log file. By default, the agent is logging to standard out. You can configure that with the log_file option.

This is not a log output that's provided by the agent. Do you know where this is coming from? Is that custom code from your application?

10

This output is my own logging.

That might be because you have looked at the wrong log file. By default, the agent is logging to standard out. You can configure that with the log_file option.

I don't completely agree here, for example, lets say I configure some junk apm URL , that log gets logged in the same file where I have logged my lines.
Even if I provide some invalid config options, that also gets logged in the same place.

Only this particular exception it didn't log and silently went ahead without any indication.

11

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.