# Node failure and recovery

**URL:** <https://discuss.elastic.co/t/node-failure-and-recovery/106065>\
**Category:** Elasticsearch\
**Created:** [November 1, 2017, 4:57pm UTC](https://discuss.elastic.co/t/node-failure-and-recovery/106065 "2017-11-01T16:57:52Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![wrinehart](https://avatars.discourse-cdn.com/v4/letter/w/ba9def/32.png) [@wrinehart](https://discuss.elastic.co/u/wrinehart)\
**Post date:** [November 1, 2017, 4:57pm UTC](https://discuss.elastic.co/t/node-failure-and-recovery/106065/1 "2017-11-01T16:57:52Z")

</div>

Howdy,

I have a cluster in Elasticsearch version 1.4.4 that is currently under-provisioned. Currently, when a node falls off the cluster the other nodes start reassigning shards from the failed nodes to the other nodes. This jacks up the heap/disk space used on the other nodes. I have the gateway.recover\_after\_nodes set so I thought that it would wait until the node had rejoined and then reinitialize the shards to the node that had failed, but that doesn't seem to be the case (i.e. when you disable shard allocation and restart a node).

Is there a setting that would wait and then reinitialize to the node that had fallen off the cluster and then rejoined?

I have these set:  
gateway.recover\_after\_nodes: 8  
gateway.expected\_nodes: 8

I am mostly just curious if I'm using these settings wrong or there is a bug in this version. I am working on upgrading the cluster to version 5.x but it won't be immediate so this would be triage in the meantime.

Thanks,  
Walt

---

<div class="post-metadata">

**Author:** ![shanec](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanec/32/4004_2.png) [@shanec](https://discuss.elastic.co/u/shanec)\
**Post date:** [November 1, 2017, 9:17pm UTC](https://discuss.elastic.co/t/node-failure-and-recovery/106065/2 "2017-11-01T21:17:24Z")

</div>

What you're looking for is delayed shard allocation ([5.x version docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/delayed-allocation.html) | [1.7 version docs](https://www.elastic.co/guide/en/elasticsearch/reference/1.7/delayed-allocation.html)). That functionality was added in 1.7.0: [https://www.elastic.co/blog/elasticsearch-1-7-0-and-1-6-1-released](https://www.elastic.co/blog/elasticsearch-1-7-0-and-1-6-1-released)

I'd recommend at least upgrading to 1.7 for a variety of reasons, not least of which includes 1.4 has multiple security vulnerabilities associated with it. But really your 5.x plan is much better.

---

<div class="post-metadata">

**Author:** ![wrinehart](https://avatars.discourse-cdn.com/v4/letter/w/ba9def/32.png) [@wrinehart](https://discuss.elastic.co/u/wrinehart)\
**Post date:** [November 2, 2017, 12:06am UTC](https://discuss.elastic.co/t/node-failure-and-recovery/106065/3 "2017-11-02T00:06:02Z")

</div>

Ah yes, I remember that's what it is called now. I definitely have clusters across all the major versions so it's easy for me to mix the settings up.

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 30, 2017, 12:06am UTC](https://discuss.elastic.co/t/node-failure-and-recovery/106065/4 "2017-11-30T00:06:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.