When using Elastic APM java agent, the JVM crashes unexpectedly. Tried using different AWS EC2 instances, happens on AMD and Graviton CPUs. Happens with JVMs with small (4GB) and large (60GB) heaps. Kibana version: 7.6.2 Elasticsearch version: 7.6.2 Java: OpenJDK Runtime Environment Corretto-17…

Thanks for the error reports. Do you have any correlation between stacks (or top of stack) and CPU architecture? Do you have any more info on when it tends to happen (how far in to the application run) and how often it happens (eg 10% of runs?). Thanks

If you can attach all the crash logs, we can do that analysis. This looks like it's not going to be easy to figure out

Thank you for having a look. It is usually between 30min and 3 hours after jvm startup. It appears to happen when we are using the app. So it happens during daytime (not at night) and more often if more users are using the app. It usually happens when doing an AWS S3 upload or download using the so…

Have you also run without the Elastic agent on the config that crashes (coretto 17) and found it stable?

Yes, that is correct. Removing the -javaagent:/home/username/elastic-apm-agent-1.35.0.jar jvm startup parameter fixed the issue and we are not experiencing any crashes.

thanks, the larger set of crash logs has a couple of crashes that show intercepts of the Elastic methods happening from AWS interceptors . Are you explicitly adding interceptors, or is that something Amazon does automatically (maybe eg for XRay tracking of errors)?

We are not explicitly adding interceptors.

And no dependencies on aws-xray-* ?

It might be worth noting that we are running on app server Payara 6.2022.2. As far as I can see they don't include the aws xray dependency either.

Could you try with this 1.35 snapshot please, it is just the latest build with the throwables no longer captured for those paths ( PR ). We suspect this is an interplay with how coretto specializes Throwable handling on AWS machines. If this stops the crashes here, we'll look at how best to apply it m…

Discuss the Elastic Stack

SIGSEGV JVM Crash

Elastic Observability APM

Jack_Shirazi (Jack Shirazi) January 31, 2023, 9:21am 22

actually we're about to release the new version with the workaround, so just test that when it's out please rather than the snapshot

Topic		Replies	Views
Java APM agent crashes JVM Corretto 17 on AWS EB Linux 2023 Elastic Observability	0	215	October 27, 2023
Java APM agent crashes JVM APM java	3	42	May 27, 2026
Getting SIGSEV with elastic apm (version 1.35.0) APM java	2	414	May 23, 2023
JVM crash originating from APM agent APM docker , java	8	1000	August 2, 2023
JVM crashed after upgrade to apm java agent 1.19 APM java	17	1532	January 18, 2021

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

SIGSEGV JVM Crash

Related topics