Shard removed from cluster - MergeException java.lang.ArrayIndexOutOfBoundsException

[Balamurali]

Hi all,

we are using ES ( 1.3.7 version ) cluster with 4 data only nodes and 3 master only nodes ,
Heap allocated for each node is 4 GB.

The issue was one shard got removed from cluster
while checking in logs , below messages are in logs

[2017-03-07 21:36:43,764][WARN ][index.merge.scheduler ] [xxxxxxxx] [yyyyyyyy][0] failed to merge
java.lang.ArrayIndexOutOfBoundsException
at org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:132)
at org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:135)
[2017-03-07 21:36:43,788][WARN ][index.engine.internal ] [xxxxxxxx] [yyyyyyyy][0] failed engine [merge exception]
[2017-03-07 21:36:43,803][DEBUG][index.shard.service ] [xxxxxxxx] [yyyyyyyy][0] state: [STARTED]->[CLOSED], reason [engine failure, message [merge exception][MergeException[java.lang.ArrayIndexOutOfBoundsException]; nested: ArrayIndexOutOfBoundsException; ]]
[2017-03-07 21:36:44,100][WARN ][cluster.action.shard ] [xxxxxxxx] [yyyyyyyy][0] sending failed shard for [yyyyyyyy][0], node[1Q0WIbVQQ4CK6Ksqetbp8w], [P], s[STARTED], indexUUID [na], reason [engine failure, message [merge exception][MergeException[java.lang.ArrayIndexOutOfBoundsException]; nested: ArrayIndexOutOfBoundsException; ]]

Due to this merge exception , shards got removed.
what is reason for this merge exception ? how to overcome these issues?

FYI: we are planning to ES upgrade to 2.x

can anyone please help us!!!!

[Balamurali]

Hi all,

any update on this?

again getting shard removal and getting same message .

[2017-03-12 23:47:52,111][WARN ][cluster.action.shard ] [XXXXXX] [YYYYYYYYY][0] received shard failed for [YYYYY][0], node[xxxx ], s[STARTED], indexUUID [na], reason [engine failure, message [merge exception][MergeException[java.lang.ArrayIndexOutOfBoundsException]; nested: ArrayIndexOutOfBoundsException; ]]

can any one help us !

[spinscale (Alexander Reelsen)]

Hey,

this is such an old version of Elasticsearch, that has its fair share of issues (for example checksums did not exist back then to detect corruption) - you might have hit https://github.com/elastic/elasticsearch/issues/7238 - which contains links to a few issues, that fixed potential index corruptions.

Please upgrade as soon as possible - it will save you a lot of pain in the future.

--Alex