IndexShardRecoveryException failed to recovery from gateway


(Shammi Kumar) #1
  1. I have 3 node elastic search cluster. Each node configuration is 12 Core / 24 GB RAM.
  2. I have 365 (days) indices. 12 shards per index (day). So total 4380+ shards
  3. I added threadpool.search.queue_size: 2500 and threadpool.bulk.queue_size: 300 in elasticsearch.yml
  4. I am performing Kibana dashboard queries from 25 visualization
  5. I got warnings to increase ulimit -v and -m with "unlimited" and in elastic search logs.
  6. Changed the ulmit configurations from default and restarted Elastic Search nodes (all 3)

Got the following error on assigning one shard out of total shards. Please provide me the solution.

[2016-09-23 16:27:04,653][WARN ][indices.cluster ] [x.x.x.x] [[security.zzzzzz.yyyyy][0]] marking and sending shard failed due to [failed recovery]
[security.zzzzzz.yyyyyy_20160421][[security.zzzzzz.yyyyyy_20160421][0]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException;
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: [security.zzzzzz.yyyyyy_20160421][[security.zzzzzz.yyyyyy_20160421][0]] EngineCreationFailureException[failed to create engine]; nested: EOFException;
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:155)
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1515)
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1499)
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:972)
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:944)
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)
... 5 more
Caused by: java.io.EOFException
at org.apache.lucene.store.InputStreamDataInput.readByte(InputStreamDataInput.java:37)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
at org.apache.lucene.store.DataInput.readLong(DataInput.java:157)
at org.elasticsearch.index.translog.Checkpoint.(Checkpoint.java:54)
at org.elasticsearch.index.translog.Checkpoint.read(Checkpoint.java:83)
at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:337)
at org.elasticsearch.index.translog.Translog.(Translog.java:179)
at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:208)
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:151)
... 11 more


(Mark Walkom) #2

What version?


(Shammi Kumar) #3

Version: 2.3.2, Build: b9e4a6a/2016-04-21T16:03:47Z, JVM: 1.8.0_92


(system) #4