# Java 6u23 and ES 0.14.2 crashing on signal (6/SIGABT) **URL:** \ **Category:** Elasticsearch\ **Created:** [January 19, 2011, 7:03pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812 "2011-01-19T19:03:26Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [January 19, 2011, 7:03pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/1 "2011-01-19T19:03:26Z")
To try and address a non-ES issue, I attempted to upgrade one node in a cluster from JDK 1.6.0\_21 to 1.6.0\_23. After several minutes of ES syncing indexes, it crashes with a SIGABT. From the wrapper logs (nothing dumped in ES logs) INFO | jvm 1 | 2011/01/19 18:37:51 | WrapperManager: Initializing... STATUS | wrapper | 2011/01/19 18:42:17 | JVM received a signal UNKNOWN (6). STATUS | wrapper | 2011/01/19 18:42:17 | JVM process is gone. ERROR | wrapper | 2011/01/19 18:42:17 | JVM exited unexpectedly. We're running CentOS 5.5 java version "1.6.0\_23" Java(TM) SE Runtime Environment (build 1.6.0\_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) Is anyone else running this combination of versions? Thanks, David ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 19, 2011, 7:05pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/2 "2011-01-19T19:05:04Z")
Several people are running ES on 1.6u23, can you give it a go without the wrapper and see if it works? Do you see a crash log by the jvm (in the work directory of where you start it)? On Wednesday, January 19, 2011 at 9:03 PM, dbenson wrote: > To try and address a non-ES issue, I attempted to upgrade one node in > a cluster from JDK 1.6.0\_21 to 1.6.0\_23. After several minutes of ES > syncing indexes, it crashes with a SIGABT. From the wrapper logs > (nothing dumped in ES logs) > > INFO | jvm 1 | 2011/01/19 18:37:51 | WrapperManager: > Initializing... > STATUS | wrapper | 2011/01/19 18:42:17 | JVM received a signal > UNKNOWN (6). > STATUS | wrapper | 2011/01/19 18:42:17 | JVM process is gone. > ERROR | wrapper | 2011/01/19 18:42:17 | JVM exited unexpectedly. > > We're running CentOS 5.5 > java version "1.6.0\_23" > Java(TM) SE Runtime Environment (build 1.6.0\_23-b05) > Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode) > > Is anyone else running this combination of versions? > > Thanks, > > David ---
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [January 19, 2011, 8:06pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/3 "2011-01-19T20:06:18Z")
I get a cash running from bin/elasticsearch as well. I am getting a crash dump, which I've uploaded to [https://gist.github.com/786752](https://gist.github.com/786752) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 19, 2011, 8:13pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/4 "2011-01-19T20:13:43Z")
Strange, it seems like it fails in the Lucene PorterStemmer, but I think its just happened to "be there". Can you remove the -XX:+AggressiveOpts added (either in the wrapper conf, or in the elasticsearch.in.sh) and see if it helps? On Wednesday, January 19, 2011 at 10:06 PM, dbenson wrote: > I get a cash running from bin/elasticsearch as well. I am getting a > crash dump, which I've uploaded to [Java crash dump ยท GitHub](https://gist.github.com/786752) ---
**Author:** ![ijuma](https://avatars.discourse-cdn.com/v4/letter/i/ed8c4c/32.png) [@ijuma](https://discuss.elastic.co/u/ijuma)\ **Post date:** [January 19, 2011, 8:44pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/5 "2011-01-19T20:44:21Z")
On Jan 19, 8:13 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Strange, it seems like it fails in the Lucene PorterStemmer, but I think its just happened to "be there". Can you remove the -XX:+AggressiveOpts added (either in the wrapper conf, or in the elasticsearch.in.sh) and see if it helps? For what is worth, using AggressiveOpts is generally a bad idea. Aside from the stability issues (that are real), it sometimes enables optimisations tailored for benchmark scores that may cause a slowdown in certain real-life workloads. Best, Ismael ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 19, 2011, 8:46pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/6 "2011-01-19T20:46:36Z")
My experience is a bit different, AgressiveOpts is a flag used for some time in quite important production systems and it works well (in my previous job)... On Wednesday, January 19, 2011 at 10:44 PM, ijuma wrote: > On Jan 19, 8:13 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Strange, it seems like it fails in the Lucene PorterStemmer, but I think its just happened to "be there". Can you remove the -XX:+AggressiveOpts added (either in the wrapper conf, or in the elasticsearch.in.sh) and see if it helps? > > For what is worth, using AggressiveOpts is generally a bad idea. Aside > from the stability issues (that are real), it sometimes enables > optimisations tailored for benchmark scores that may cause a slowdown > in certain real-life workloads. > > Best, > Ismael ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 19, 2011, 8:48pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/7 "2011-01-19T20:48:01Z")
Grr, hit send prematurely. Where have you seen this flag having bad affect? On Wednesday, January 19, 2011 at 10:46 PM, Shay Banon wrote: > My experience is a bit different, AgressiveOpts is a flag used for some time in quite important production systems and it works well (in my previous job)... > On Wednesday, January 19, 2011 at 10:44 PM, ijuma wrote: > > > On Jan 19, 8:13 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Strange, it seems like it fails in the Lucene PorterStemmer, but I think its just happened to "be there". Can you remove the -XX:+AggressiveOpts added (either in the wrapper conf, or in the elasticsearch.in.sh) and see if it helps? > > > > For what is worth, using AggressiveOpts is generally a bad idea. Aside > > from the stability issues (that are real), it sometimes enables > > optimisations tailored for benchmark scores that may cause a slowdown > > in certain real-life workloads. > > > > Best, > > Ismael ---
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [January 19, 2011, 9:05pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/8 "2011-01-19T21:05:45Z")
Removing -XX:+AggressiveOpts did the trick. Any idea as to why this may have been the culprit? Or perhaps I should stay on 1.6u21, since u23 didn't address my other issue. Thanks, David ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 19, 2011, 9:07pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/9 "2011-01-19T21:07:37Z")
Well, its good to know. 1.6u23 is new, so this flag might cause it problems, I will remove it from being applied by default (in both the wrapper repo and the default elasticsearch.in.sh). Not sure why it was caused though... I suggest you stay with 1.6u23 without the flag, even if it doesn't fix the issue. On Wednesday, January 19, 2011 at 11:05 PM, dbenson wrote: > Removing -XX:+AggressiveOpts did the trick. > > Any idea as to why this may have been the culprit? Or perhaps I should > stay on 1.6u21, since u23 didn't address my other issue. > > Thanks, > > David ---
**Author:** ![ijuma](https://avatars.discourse-cdn.com/v4/letter/i/ed8c4c/32.png) [@ijuma](https://discuss.elastic.co/u/ijuma)\ **Post date:** [January 19, 2011, 9:17pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/10 "2011-01-19T21:17:45Z")
On Jan 19, 8:48 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Grr, hit send prematurely. Where have you seen this flag having bad affect? AggressiveOpts enables a set of optimizations. At some point (not sure if it still does), it would enable a special HashMap that treated Integers specially because it made some benchmark faster and according to Doug Lea this was not a general win (and why it was never enabled without the switch). At other points, it enabled things that were not ready and would cause crashes (like escape analysis, StringBuilder.toString optimisation and so on). I follow HotSpot pretty closely and I see crashes being fixed for optimizations that are enabled by AggressiveOpts more often than I feel comfortable with. With the policy of updating JDK6 with a new HotSpot every few releases, it's risky enough to use it without AggressiveOpts in my opinion. I can probably name 4 or 5 major JIT/GC bugs (that would lead to crashes or miscompilation, were easy to trigger and/or affected major open-source projects) without AggressiveOpts in the JDK 6 series. Best, Ismael ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 19, 2011, 10:21pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/11 "2011-01-19T22:21:47Z")
Right, well it used to work pretty well up to 1.6u23 :). I pushed a change to not use it by default :). On Wednesday, January 19, 2011 at 11:17 PM, ijuma wrote: > On Jan 19, 8:48 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Grr, hit send prematurely. Where have you seen this flag having bad affect? > > AggressiveOpts enables a set of optimizations. At some point (not sure > if it still does), it would enable a special HashMap that treated > Integers specially because it made some benchmark faster and according > to Doug Lea this was not a general win (and why it was never enabled > without the switch). At other points, it enabled things that were not > ready and would cause crashes (like escape analysis, > StringBuilder.toString optimisation and so on). > > I follow HotSpot pretty closely and I see crashes being fixed for > optimizations that are enabled by AggressiveOpts more often than I > feel comfortable with. With the policy of updating JDK6 with a new > HotSpot every few releases, it's risky enough to use it without > AggressiveOpts in my opinion. I can probably name 4 or 5 major JIT/GC > bugs (that would lead to crashes or miscompilation, were easy to > trigger and/or affected major open-source projects) without > AggressiveOpts in the JDK 6 series. > > Best, > Ismael ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [March 9, 2011, 9:44pm UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/12 "2011-03-09T21:44:20Z")
FYI, I just hit this issue on one out of three nodes (and repeatedly on that node) after an upgrade to 15.2 from 14.2. Thought it was worth mentioning, since it only effects a single node and it started after the upgrade. The stack trace looks very similar to the one captured by Ismael. I am upgrading to the latest service wrapper, now. I expect that to resolve things. Thanks, Paul On Jan 19, 3:21 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Right, well it used to work pretty well up to 1.6u23 :). I pushed a change to not use it by default :). > > On Wednesday, January 19, 2011 at 11:17 PM, ijuma wrote: > > > On Jan 19, 8:48 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > Grr, hit send prematurely. Where have you seen this flag having bad affect? > > > AggressiveOpts enables a set of optimizations. At some point (not sure > > if it still does), it would enable a special HashMap that treated > > Integers specially because it made some benchmark faster and according > > to Doug Lea this was not a general win (and why it was never enabled > > without the switch). At other points, it enabled things that were not > > ready and would cause crashes (like escape analysis, > > StringBuilder.toString optimisation and so on). > > > I follow HotSpot pretty closely and I see crashes being fixed for > > optimizations that are enabled by AggressiveOpts more often than I > > feel comfortable with. With the policy of updating JDK6 with a new > > HotSpot every few releases, it's risky enough to use it without > > AggressiveOpts in my opinion. I can probably name 4 or 5 major JIT/GC > > bugs (that would lead to crashes or miscompilation, were easy to > > trigger and/or affected major open-source projects) without > > AggressiveOpts in the JDK 6 series. > > > Best, > > Ismael ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:10am UTC](https://discuss.elastic.co/t/java-6u23-and-es-0-14-2-crashing-on-signal-6-sigabt/3812/13 "2017-07-06T04:10:24Z")