# SIGSEGV JVM Crash **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [January 10, 2023, 9:27am UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812 "2023-01-10T09:27:57Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![mafaul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mafaul/32/115754_2.png) [@mafaul](https://discuss.elastic.co/u/mafaul)\ **Post date:** [January 10, 2023, 9:27am UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/1 "2023-01-10T09:27:57Z")
When using Elastic APM java agent, the JVM crashes unexpectedly. Tried using different AWS EC2 instances, happens on AMD and Graviton CPUs. Happens with JVMs with small (4GB) and large (60GB) heaps. **Kibana version** : 7.6.2 **Elasticsearch version** : 7.6.2 **Java** : OpenJDK Runtime Environment Corretto-17.0.5.8.1 64it (x86 and Aarch) **APM Agent language and version** : Java 1.35.0 **OS** : Amazon Linux 2 (x86 and Aarch) I've attached 3 JVM crash files below as replies. I have about 15 more. Unfortunately I cannot reproduce the issue on demand, to me it appears to happen randomly. It look a bit like [Segmentation fault when attaching apm-agent-java to adopt jdk 11 · Issue #864 · elastic/apm-agent-java · GitHub](https://github.com/elastic/apm-agent-java/issues/864), but with Java 17 ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [January 10, 2023, 11:42am UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/5 "2023-01-10T11:42:32Z")
Thanks for the error reports. Do you have any correlation between stacks (or top of stack) and CPU architecture? Do you have any more info on when it tends to happen (how far in to the application run) and how often it happens (eg 10% of runs?). Thanks ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [January 10, 2023, 12:22pm UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/6 "2023-01-10T12:22:36Z")
If you can attach all the crash logs, we can do that analysis. This looks like it's not going to be easy to figure out ---
**Author:** ![mafaul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mafaul/32/115754_2.png) [@mafaul](https://discuss.elastic.co/u/mafaul)\ **Post date:** [January 10, 2023, 12:24pm UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/7 "2023-01-10T12:24:45Z")
Thank you for having a look. It is usually between 30min and 3 hours after jvm startup. It appears to happen when we are using the app. So it happens during daytime (not at night) and more often if more users are using the app. It usually happens when doing an AWS S3 upload or download using the software.amazon.awssdk s3 (version 2.19.2). It happens on Aarch (Graviton 2) and X86 64 bit (AMD Epyc) cpu architectures. This issue did not happen with the java apm agent version 1.20.0 on Java 11.0.16.1. Crash logs: [hs\_err - Google Drive](https://drive.google.com/drive/folders/1NrJApfTqYqK0CfoHA9mwxQteowDwpmyt?usp=sharing) ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [January 10, 2023, 12:39pm UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/8 "2023-01-10T12:39:27Z")
Have you also run **without** the Elastic agent on the config that crashes (coretto 17) and found it stable? ---
**Author:** ![mafaul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mafaul/32/115754_2.png) [@mafaul](https://discuss.elastic.co/u/mafaul)\ **Post date:** [January 10, 2023, 12:46pm UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/9 "2023-01-10T12:46:41Z")
Yes, that is correct. Removing the > -javaagent:/home/username/elastic-apm-agent-1.35.0.jar jvm startup parameter fixed the issue and we are not experiencing any crashes. ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [January 10, 2023, 1:46pm UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/10 "2023-01-10T13:46:43Z")
thanks, the larger set of crash logs has a couple of crashes that show intercepts of the Elastic methods happening from [AWS interceptors](https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/core/interceptor/ExecutionInterceptor.html). Are you explicitly adding interceptors, or is that something Amazon does automatically (maybe eg for XRay tracking of errors)? ---
**Author:** ![mafaul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mafaul/32/115754_2.png) [@mafaul](https://discuss.elastic.co/u/mafaul)\ **Post date:** [January 10, 2023, 1:53pm UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/11 "2023-01-10T13:53:33Z")
We are not explicitly adding interceptors. ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [January 10, 2023, 2:25pm UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/12 "2023-01-10T14:25:01Z")
And no dependencies on aws-xray-\* ? ---
**Author:** ![mafaul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mafaul/32/115754_2.png) [@mafaul](https://discuss.elastic.co/u/mafaul)\ **Post date:** [January 11, 2023, 6:25am UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/14 "2023-01-11T06:25:51Z")
It might be worth noting that we are running on app server Payara 6.2022.2. As far as I can see they don't include the aws xray dependency either. ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [January 11, 2023, 1:06pm UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/15 "2023-01-11T13:06:19Z")
Could you try with this [1.35 snapshot](https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-2958/1/artifact/src/github.com/elastic/apm-agent-java/elastic-apm-agent/target/elastic-apm-agent-1.35.1-SNAPSHOT.jar) please, it is just the latest build with the throwables no longer captured for those paths ([PR](https://github.com/elastic/apm-agent-java/pull/2958)). We suspect this is an interplay with how coretto specializes Throwable handling on AWS machines. If this stops the crashes here, we'll look at how best to apply it more generically ---
**Author:** ![mafaul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mafaul/32/115754_2.png) [@mafaul](https://discuss.elastic.co/u/mafaul)\ **Post date:** [January 11, 2023, 2:26pm UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/16 "2023-01-11T14:26:44Z")
Sure, will give it a go and revert back tomorrow. ---
**Author:** ![mafaul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mafaul/32/115754_2.png) [@mafaul](https://discuss.elastic.co/u/mafaul)\ **Post date:** [January 12, 2023, 1:01pm UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/17 "2023-01-12T13:01:39Z")
It works ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [January 12, 2023, 3:54pm UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/18 "2023-01-12T15:54:38Z")
Thanks! We'll produce a more complete PR and include that in an upcoming release ---
**Author:** ![mafaul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mafaul/32/115754_2.png) [@mafaul](https://discuss.elastic.co/u/mafaul)\ **Post date:** [January 13, 2023, 7:45am UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/19 "2023-01-13T07:45:35Z")
Than you. I appreciate it ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [January 13, 2023, 2:47pm UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/20 "2023-01-13T14:47:33Z")
Just for completeness, please test the [updated snapshot](https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-2958/3/artifact/src/github.com/elastic/apm-agent-java/elastic-apm-agent/target/elastic-apm-agent-1.35.1-SNAPSHOT.jar) which looks specifically for a corretto JVM and avoids the capture only in that case ---
**Author:** ![mafaul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mafaul/32/115754_2.png) [@mafaul](https://discuss.elastic.co/u/mafaul)\ **Post date:** [January 31, 2023, 8:52am UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/21 "2023-01-31T08:52:17Z")
Will do, apologies for the delay. It will be tested tomorrow ---
**Author:** ![Jack\_Shirazi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_shirazi/32/91641_2.png) [@Jack\_Shirazi](https://discuss.elastic.co/u/Jack_Shirazi)\ **Post date:** [January 31, 2023, 9:21am UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/22 "2023-01-31T09:21:57Z")
actually we're about to release the new version with the workaround, so just test that when it's out please rather than the snapshot ---
**Author:** ![mafaul](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mafaul/32/115754_2.png) [@mafaul](https://discuss.elastic.co/u/mafaul)\ **Post date:** [February 4, 2023, 5:01am UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/23 "2023-02-04T05:01:37Z")
v1.36.0 works perfectly, thank you. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 25, 2023, 1:02am UTC](https://discuss.elastic.co/t/sigsegv-jvm-crash/322812/24 "2023-02-25T01:02:35Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.