# JVM crash with JRE 7 and profiling\_inferred\_spans\_enabled **URL:** \ **Category:** APM\ **Tags:** java\ **Created:** [February 19, 2021, 2:41pm UTC](https://discuss.elastic.co/t/jvm-crash-with-jre-7-and-profiling-inferred-spans-enabled/264871 "2021-02-19T14:41:46Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Charles\_Porter](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_porter/32/84135_2.png) [@Charles\_Porter](https://discuss.elastic.co/u/Charles_Porter)\ **Post date:** [February 19, 2021, 2:41pm UTC](https://discuss.elastic.co/t/jvm-crash-with-jre-7-and-profiling-inferred-spans-enabled/264871/1 "2021-02-19T14:41:46Z")
Hello. We're currently testing out Elastic APM in our java-based test environment for an application my team supports. One of the application components runs in a Java 7 environment. We initially encountered issue [1583](https://github.com/elastic/apm-agent-java/issues/1583), and applied the suggested `-XX:CompileCommand` to work around the issue. However, whenever we set `profiling_inferred_spans_enabled` to true, shortly afterward, the JVM crashes. We are interested in being able to collect the extra span information provided `profiling_inferred_spans_enabled`, and are hoping there is a way to work through this. Thank you. **OS Version:** CentOS 6.10 **Java version:** Oracle JRE 7.0\_80-b15 **APM Agent version** : apm-java-agent 1.21.0 The apm-agent attaches to the apache tomcat process via the -javaagent flag. Default settings. **elasticapm.properties:** `use_path_as_transaction_name=true` `server_urls=https://xxx.com/apm` `verify_server_sert=false` `api_key=xxx` **top of the hs\_err\_pid log** : ```auto # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007ffa751cb08b, pid=13985, tid=140712926508800 # # JRE version: Java(TM) SE Runtime Environment (7.0_80-b15) (build 1.7.0_80-b15) # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode linux-amd64 compressed oops) # Problematic frame: # V [libjvm.so+0x6ec08b] JvmtiEnvBase::get_stack_trace(JavaThread*, int, int, _jvmtiFrameInfo*, int*)+0x21b # # Core dump written. Default location: /apps/tomcat8/sbtool/bin/core or core.13985 # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x00007ffa7042f800): JavaThread "Unknown thread" [_thread_blocked, id=14101, stack(0x00007ffa47eff000,0x00007ffa48000000)] siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000f60115 Registers: RAX=0x0000000000f600f5, RBX=0x00007ffa7012ade0, RCX=0x0000003e2b8182a0, RDX=0x00007ffa7010f4f8 RSP=0x00007ffa47ff8c80, RBP=0x00007ffa47ff9300, RSI=0x00007ffa7010f4f0, RDI=0x0000000000000000 R8 =0x0000000000000000, R9 =0x0000000000000800, R10=0x0000000000000000, R11=0x0000000000000246 R12=0x00000000f4dd59a8, R13=0x0000000002782240, R14=0x00007ffa7010f4f0, R15=0x00007ffa58664d38 RIP=0x00007ffa751cb08b, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000004 TRAPNO=0x000000000000000e ``` ---
**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\ **Post date:** [February 19, 2021, 3:16pm UTC](https://discuss.elastic.co/t/jvm-crash-with-jre-7-and-profiling-inferred-spans-enabled/264871/2 "2021-02-19T15:16:37Z")
Hi @Charles_Porter , I am sorry for the inconvenience, and thanks for reporting this issue. Do you think you could send us the full crash report (make sure that any sensitive environment variable or JVM command line parameter is removed) ? Also, what is the frequency of the crashes that you observed ? From the past JVM crashes related to `JvmtiEnvBase::get_stack_trace` we managed to work-around those stability issues on some JVMs by using the `async_profiler_safe_mode=63` configuration parameter. This (yet un-documented) configuration allows to make async-profiler avoid collecting some stack traces for extra safety. Please try with it and tell us if it makes any difference. Depending on the result, some trial & error might be required to properly tune the value and identify what is causing this within async-profiler. Also, it's completely unrelated but there is a small typo in the config you have written here, `verify_server_sert` should probably be replaced by `verify_server_cert`. ---
**Author:** ![Charles\_Porter](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_porter/32/84135_2.png) [@Charles\_Porter](https://discuss.elastic.co/u/Charles_Porter)\ **Post date:** [February 19, 2021, 7:02pm UTC](https://discuss.elastic.co/t/jvm-crash-with-jre-7-and-profiling-inferred-spans-enabled/264871/3 "2021-02-19T19:02:47Z")
Hi Sylvain\_Juge, Thank you for responding. Using `async_profiler_safe_mode=63` seems to have improved things substantially. No JVM crashes, yet. I've submitted the full `hs_err_pid.log` through my Elastic Sales contact. Also, thanks for catching the typo. My eyes are clearly not what the used to be. 🙃 Thank you. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [February 21, 2021, 11:22am UTC](https://discuss.elastic.co/t/jvm-crash-with-jre-7-and-profiling-inferred-spans-enabled/264871/4 "2021-02-21T11:22:33Z")
Thanks for the update @Charles_Porter When you get the chance, please try out [this bugfix](https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/async-profiler-fix/1/artifact/src/github.com/elastic/apm-agent-java/elastic-apm-agent/target/elastic-apm-agent-1.21.1-SNAPSHOT.jar) without the `async_profiler_safe_mode` setting and see if the problem is resolved. This snapshot contains the proposed fix for Async Profiler. Thanks! ---
**Author:** ![Charles\_Porter](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/charles_porter/32/84135_2.png) [@Charles\_Porter](https://discuss.elastic.co/u/Charles_Porter)\ **Post date:** [February 22, 2021, 7:02pm UTC](https://discuss.elastic.co/t/jvm-crash-with-jre-7-and-profiling-inferred-spans-enabled/264871/5 "2021-02-22T19:02:38Z")
Thank you, @Eyal_Koren The bugfix appears to be successful. After installing, I have disabled `async_profiler_safe_mode` and restarted the application. Simulated workloads have not produced any JVM crashes. Thank you very much for helping. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [February 23, 2021, 7:02am UTC](https://discuss.elastic.co/t/jvm-crash-with-jre-7-and-profiling-inferred-spans-enabled/264871/6 "2021-02-23T07:02:38Z")
Awesome! Thanks for reporting back! I am also running load tests during the past ~48 hours with that fix on a setup that previously reproduced the issue you reported, with quite intensive load on async profiler (very frequent sampling) and it looks good. For now, you can continue using this snapshot. We will make sure to include this fix in our next release. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 16, 2021, 3:03am UTC](https://discuss.elastic.co/t/jvm-crash-with-jre-7-and-profiling-inferred-spans-enabled/264871/7 "2021-03-16T03:03:09Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.