Error in APM agent during startup

jibitgeorge · July 3, 2020, 4:03pm

Kibana version: 7.6.2

Elasticsearch version: 7.6.2

APM Server version: 7.6.2

APM Agent language and version: Java 1.12.0

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version?

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

NO

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I have a java APM agent attached to a springBoot application. I can see the application listed in APM server but none of the transactions are getting reflected in APM dashboard. This is the issue I'm facing.

I could see the JVM related information in APM dashboard. But the problem is transactions are not getting reflected, even after I send multiple transaction in my application.

Below are the log, in which you can see one error and thereafter same set of statements are repeatedly printed.

Picked up JAVA_TOOL_OPTIONS: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar
2020-07-03 05:40:58.638 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Found Kubernetes pod UID: bf3d1859-bcef-11ea-b38e-42010a9600b5
2020-07-03 05:40:58.640 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - container ID is a95c388bbef65dd621949ba8631cf1f620c225e31460331621ac9e0d1656dd8e
2020-07-03 05:40:58.847 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.12.0 as Inbound-processor-oru on Java 1.8.0_171 (Oracle Corporation) Linux 4.14.94+
2020-07-03 05:40:58.848 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - service_name: 'Inbound-processor-oru' (source: Environment Variables)
2020-07-03 05:40:58.848 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - environment: 'prod' (source: Environment Variables)
2020-07-03 05:40:58.848 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - capture_body: 'ALL' (source: Environment Variables)
2020-07-03 05:40:58.848 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - server_urls: 'http://common-elk-apm-server.default.svc.cluster.local:8200' (source: Environment Variables)
2020-07-03 05:40:58.849 [main] WARN co.elastic.apm.agent.configuration.StartupInfo - Detected usage of an old configuration key: 'server_url'. Please use 'server_urls' instead.
2020-07-03 05:40:58.849 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - log_level: 'DEBUG' (source: Environment Variables)
2020-07-03 05:40:58.849 [main] WARN co.elastic.apm.agent.configuration.StartupInfo - To enable all features and to increase startup times, please configure application_packages
2020-07-03 05:40:58.937 [main] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Deferring initialization of JMX metric tracking until capture_jmx_metrics is set.
2020-07-03 05:40:58.958 [apm-server-healthcheck] DEBUG co.elastic.apm.agent.report.ApmServerHealthChecker - Starting healthcheck to http://common-elk-apm-server.default.svc.cluster.local:8200/
2020-07-03 05:40:58.958 [apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Reloading configuration from APM Server http://common-elk-apm-server.default.svc.cluster.local:8200/config/v1/agents
2020-07-03 05:40:59.146 [apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Central configuration is disabled. Set kibana.enabled: true in your APM Server configuration.
2020-07-03 05:40:59.149 [apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Scheduling next remote configuration reload in 300s
2020-07-03 05:40:59.149 [apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2020-03-26T04:23:50Z",  "build_sha": "8827b43d24eaf42391ea815bb9fa12bdef5d6bce",  "version": "7.6.2"}
2020-07-03 05:40:59.152 [apm-server-healthcheck] DEBUG co.elastic.apm.agent.report.ApmServerHealthChecker - APM server http://common-elk-apm-server.default.svc.cluster.local:8200/ version is: 7.6.2
.
.
.
.


2020-07-03 05:41:26.643 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Type match for instrumentation SpringServiceNameInstrumentation: hasSuperType(erasure(name(equals(org.springframework.web.context.WebApplicationContext)))) matches class org.springframework.web.context.support.GenericWebApplicationContext
2020-07-03 05:41:26.647 [main] INFO co.elastic.apm.agent.bci.bytebuddy.ErrorLoggingListener - org.springframework.web.context.support.GenericWebApplicationContext refers to a missing class
java.lang.IllegalStateException: Cannot resolve type description for javax.servlet.ServletContext
	at co.elastic.apm.agent.shaded.bytebuddy.pool.TypePool$Resolution$Illegal.resolve(TypePool.java:159)
	at co.elastic.apm.agent.shaded.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType.toErasure(TypePool.java:6241)
	at co.elastic.apm.agent.shaded.bytebuddy.pool.TypePool$Default$LazyTypeDescription$GenericTypeToken$Resolution$Raw$RawAnnotatedType.of(TypePool.java:3412)
	at co.elastic.apm.agent.shaded.bytebuddy.pool.TypePool$Default$LazyTypeDescription$GenericTypeToken$Resolution$Raw.resolveReturnType(TypePool.java:3302)
.
.
.
.

2020-07-03 06:47:38.852 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Flushing 1698 uncompressed 737 compressed bytes
2020-07-03 06:47:58.840 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICS event (sequence 266)
2020-07-03 06:47:58.840 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://common-elk-apm-server.default.svc.cluster.local:8200/intake/v2/events
2020-07-03 06:47:58.854 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10s
2020-07-03 06:48:08.854 [apm-request-timeout-timer] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Request flush because the request timeout occurred
2020-07-03 06:48:08.855 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving FLUSH event (sequence 267)
2020-07-03 06:48:08.855 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Flushing 1698 uncompressed 739 compressed bytes
2020-07-03 06:48:28.842 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICS event (sequence 268)
2020-07-03 06:48:28.842 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://common-elk-apm-server.default.svc.cluster.local:8200/intake/v2/events
2020-07-03 06:48:28.844 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Scheduling request timeout in 10s

Sylvain_Juge · July 7, 2020, 7:00am

Hi @jibitgeorge,

The 1.12.0 version is a bit old, could you try to update to the latest version 1.17.0 to see if there is still an issue ?

Also, which version of Spring boot are you using ? Does the application is servlet-based or relies on another framework like Webflux (not yet supported) ?

For the exception we can see in the partial logs you provided, it should be possible to disable spring-service-name instrumentation by setting disable_instrumentations=spring-service-name, and that would also require to set service_name explicitly as it won't be detected automatically from Spring configuration.

jibitgeorge · July 7, 2020, 7:32am

@Sylvain_Juge Thanks for the reply

I will try by upgrading the agent version.

To answer your question, I'm using Sprint Boot 2.0.1

system · July 28, 2020, 3:33am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.