JAVA APM Agent latest version is crashing the jvm of the application

Elastic Observability APM
1

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version: 8.17.1

Elasticsearch version: 8.17.1

APM Server version:

APM Agent language and version: java, open jdk JRE version: OpenJDK Runtime Environment Zulu17.50+20-SA (17.0.11+9) (build 17.0.11+9-LTS)

Java VM: OpenJDK 64-Bit Server VM Zulu17.50+20-SA (17.0.11+9-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version?

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

2

Please find the logs ...

JRE version: OpenJDK Runtime Environment Zulu17.50+20-SA (17.0.11+9) (build 17.0.11+9-LTS)

Java VM: OpenJDK 64-Bit Server VM Zulu17.50+20-SA (17.0.11+9-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)

Problematic frame:

J 78237 c2 co.elastic.apm.agent.report.CountingOutputStream.write([BII)V (22 bytes) @ 0x00007ff5f5b1fcd8 [0x00007ff5f5b1fca0+0x0000000000000038]

Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump

3

Hi, your application currently have at least two things that could make it unstable:

  • -Xverify:none indicates that bytecode verification is not enforced, thus invalid bytecode could potentially trigger issues when the C2 compiler executes (usually after a bit of time when the application code gets "warm").
  • you have the AppDynamic agent configured, and running with more than one instrumentation agent is not recommended nor supported .

The instrumentation agent should not trigger directly JVM crashes as it just modifies application bytecode, unless the sampling profiler is enabled (which is not in your case).
It can however trigger some existing bugs in the JVM, in particular in GC or C2 compiler areas that were not triggered previously. Updating to the latest JVM version can also be beneficial here (Zulu latest is 17.0.15, so not that far from the one your are using).

Also, I see that the application seems to be related to SoftwareAG, which is not supported out of the box, so you might miss some transactions and spans. If that's the case I would suggest to contact their support and ask if they provide an OpenTelemetry integration for their application. If that's the case then using the Java EDOT agent or the OpenTelemetry instrumentation agent could be alternative options.

4

Thank you so much for your guidance. let me check on these steps and revert to you.

regards
AK

5

In addition, the JVM version that you are using is affected by this JVM bug https://bugs.openjdk.org/browse/JDK-8322726 which happens to be easily triggered when using our instrumentation agent. So updating the JVM to 17.0.13 or later should fix it.