# IndexShardRecoveryException failed to recovery from gateway

**URL:** <https://discuss.elastic.co/t/indexshardrecoveryexception-failed-to-recovery-from-gateway/61370>\
**Category:** Elasticsearch\
**Created:** [September 23, 2016, 11:11am UTC](https://discuss.elastic.co/t/indexshardrecoveryexception-failed-to-recovery-from-gateway/61370 "2016-09-23T11:11:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shammi\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@Shammi\_Kumar](https://discuss.elastic.co/u/Shammi_Kumar)\
**Post date:** [September 23, 2016, 11:11am UTC](https://discuss.elastic.co/t/indexshardrecoveryexception-failed-to-recovery-from-gateway/61370/1 "2016-09-23T11:11:43Z")

</div>

1. I have 3 node elastic search cluster. Each node configuration is 12 Core / 24 GB RAM.
2. I have 365 (days) indices. 12 shards per index (day). So total 4380+ shards
3. I added threadpool.search.queue\_size: 2500 and threadpool.bulk.queue\_size: 300 in elasticsearch.yml
4. I am performing Kibana dashboard queries from 25 visualization
5. I got warnings to increase ulimit -v and -m with "unlimited" and in elastic search logs.
6. Changed the ulmit configurations from default and restarted Elastic Search nodes (all 3)

Got the following error on assigning one shard out of total shards. Please provide me the solution.

[2016-09-23 16:27:04,653][WARN][indices.cluster] [x.x.x.x] [[security.zzzzzz.yyyyy][0]] marking and sending shard failed due to [failed recovery]  
[security.zzzzzz.yyyyyy\_20160421][[security.zzzzzz.yyyyyy\_20160421][0]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException;  
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:250)  
at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:56)  
at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:129)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:745)  
Caused by: [security.zzzzzz.yyyyyy\_20160421][[security.zzzzzz.yyyyyy\_20160421][0]] EngineCreationFailureException[failed to create engine]; nested: EOFException;  
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:155)  
at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)  
at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1515)  
at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1499)  
at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:972)  
at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:944)  
at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:241)  
... 5 more  
Caused by: java.io.EOFException  
at org.apache.lucene.store.InputStreamDataInput.readByte(InputStreamDataInput.java:37)  
at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)  
at org.apache.lucene.store.DataInput.readLong(DataInput.java:157)  
at org.elasticsearch.index.translog.Checkpoint.(Checkpoint.java:54)  
at org.elasticsearch.index.translog.Checkpoint.read(Checkpoint.java:83)  
at org.elasticsearch.index.translog.Translog.recoverFromFiles(Translog.java:337)  
at org.elasticsearch.index.translog.Translog.(Translog.java:179)  
at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:208)  
at org.elasticsearch.index.engine.InternalEngine.(InternalEngine.java:151)  
... 11 more

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 25, 2016, 7:35am UTC](https://discuss.elastic.co/t/indexshardrecoveryexception-failed-to-recovery-from-gateway/61370/2 "2016-09-25T07:35:58Z")

</div>

What version?

---

<div class="post-metadata">

**Author:** ![Shammi\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@Shammi\_Kumar](https://discuss.elastic.co/u/Shammi_Kumar)\
**Post date:** [September 25, 2016, 5:13pm UTC](https://discuss.elastic.co/t/indexshardrecoveryexception-failed-to-recovery-from-gateway/61370/3 "2016-09-25T17:13:11Z")

</div>

Version: 2.3.2, Build: b9e4a6a/2016-04-21T16:03:47Z, JVM: 1.8.0\_92

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:17pm UTC](https://discuss.elastic.co/t/indexshardrecoveryexception-failed-to-recovery-from-gateway/61370/4 "2017-07-05T22:17:25Z")

</div>