Cluster becomes unresponsive and cluster health becomes yellow

Elastic Stack Elasticsearch
1

Stack trace on the node where shard is present :

[2016-07-07 17:46:57,374][WARN ][index.merge.scheduler ] [elasticsearch-4] [googlix][62] failed to merge
java.io.EOFException: seek past EOF: MMapIndexInput(path="/var/lib/es/googlix-v2/nodes/0/indices/googlix/62/index/_jxzn7_es090_0.pos")
at org.apache.lucene.store.ByteBufferIndexInput.seek(ByteBufferIndexInput.java:174)
at org.apache.lucene.codecs.lucene41.ForUtil.skipBlock(ForUtil.java:220)
at org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.skipPositions(Lucene41PostingsReader.java:895)
at org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.nextPosition(Lucene41PostingsReader.java:925)
at org.apache.lucene.codecs.MappingMultiDocsAndPositionsEnum.nextPosition(MappingMultiDocsAndPositionsEnum.java:120)
at org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:118)
at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:164)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4146)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3743)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
[2016-07-07 17:46:57,383][WARN ][index.engine.internal ] [elasticsearch-4] [googlix][62] failed engine
org.apache.lucene.index.MergePolicy$MergeException: java.io.EOFException: seek past EOF: MMapIndexInput(path="/var/lib/es/googlix-v2/nodes/0/indices/googlix/62/index/_jxzn7_es090_0.pos")
at org.elasticsearch.index.merge.scheduler.ConcurrentMergeSchedulerProvider$CustomConcurrentMergeScheduler.handleMergeException(ConcurrentMergeSchedulerProvider.java:109)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:518)
Caused by: java.io.EOFException: seek past EOF: MMapIndexInput(path="/var/lib/es/googlix-v2/nodes/0/indices/googlix/62/index/_jxzn7_es090_0.pos")
at org.apache.lucene.store.ByteBufferIndexInput.seek(ByteBufferIndexInput.java:174)
at org.apache.lucene.codecs.lucene41.ForUtil.skipBlock(ForUtil.java:220)
at org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.skipPositions(Lucene41PostingsReader.java:895)
at org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.nextPosition(Lucene41PostingsReader.java:925)
at org.apache.lucene.codecs.MappingMultiDocsAndPositionsEnum.nextPosition(MappingMultiDocsAndPositionsEnum.java:120)
at org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:118)
at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:164)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4146)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3743)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:107)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
[2016-07-07 17:46:57,398][WARN ][cluster.action.shard ] [elasticsearch-4] [googlix][62] sending failed shard for [googlix][62], node[uRmgTqArSu6H5rDmCYRQQA], [P], s[STARTED], indexUUID [na], reason [engine failure, messa
ge [MergeException[java.io.EOFException: seek past EOF: MMapIndexInput(path="/var/lib/es/googlix-v2/nodes/0/indices/googlix/62/index/_jxzn7_es090_0.pos")]; nested: EOFException[seek past EOF: MMapIndexInput(path="/var/lib/es/g
ooglix-v2/nodes/0/indices/googlix/62/index/_jxzn7_es090_0.pos")]; ]]

2

Stack trace from node where the shard is not present :

[2016-07-07 17:46:57,401][WARN ][cluster.action.shard ] [Jonothon Starsmore] [googlix][62] received shard failed for [googlix][62], node[uRmgTqArSu6H5rDmCYRQQA], [P], s[STARTED], indexUUID [na], reason [engine failure, message [MergeException[java.io.EOFException: seek past EOF: MMapIndexInput(path="/var/lib/es/googlix-v2/nodes/0/indices/googlix/62/index/_jxzn7_es090_0.pos")]; nested: EOFException[seek past EOF: MMapIndexInput(path="/var/lib/es/googlix-v2/nodes/0/indices/googlix/62/index/_jxzn7_es090_0.pos")]; ]]

3

Getting this too in logs when trying to search:

[2016-07-07 23:29:19,420][DEBUG][action.search.type ] [elasticsearch-3] All shards failed for phase: [query_fetch]

4

What version are you on?

5

1.1.1 (I know this is ancient)