# How to recover a failed data node in Production cluster? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 26, 2017, 5:20pm UTC](https://discuss.elastic.co/t/how-to-recover-a-failed-data-node-in-production-cluster/76547 "2017-02-26T17:20:24Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![ImdotnetJunkie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/imdotnetjunkie/32/15514_2.png) [@ImdotnetJunkie](https://discuss.elastic.co/u/ImdotnetJunkie)\ **Post date:** [February 26, 2017, 5:20pm UTC](https://discuss.elastic.co/t/how-to-recover-a-failed-data-node-in-production-cluster/76547/1 "2017-02-26T17:20:24Z")
What steps must to be performed to recover a failed node in a cluster? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 26, 2017, 8:37pm UTC](https://discuss.elastic.co/t/how-to-recover-a-failed-data-node-in-production-cluster/76547/2 "2017-02-26T20:37:51Z")
You should just be able to restart it and it'll rejoin. Though that may depend on the failure mode. ---
**Author:** ![ImdotnetJunkie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/imdotnetjunkie/32/15514_2.png) [@ImdotnetJunkie](https://discuss.elastic.co/u/ImdotnetJunkie)\ **Post date:** [February 27, 2017, 4:07am UTC](https://discuss.elastic.co/t/how-to-recover-a-failed-data-node-in-production-cluster/76547/3 "2017-02-27T04:07:09Z")
What if unassigned\_shards: \> 0 OR initializing\_shards: \> 0 OR in general, Cluster health is RED or Yellow ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 27, 2017, 4:09am UTC](https://discuss.elastic.co/t/how-to-recover-a-failed-data-node-in-production-cluster/76547/4 "2017-02-27T04:09:01Z")
If it's initialising then just wait for everything to finish. If they're unassigned then, it depends. What version are you on? ---
**Author:** ![ImdotnetJunkie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/imdotnetjunkie/32/15514_2.png) [@ImdotnetJunkie](https://discuss.elastic.co/u/ImdotnetJunkie)\ **Post date:** [February 27, 2017, 4:10am UTC](https://discuss.elastic.co/t/how-to-recover-a-failed-data-node-in-production-cluster/76547/5 "2017-02-27T04:10:27Z")
We are still on old version 1.7.5 though we are planning to upgrade. This is our current status: cluster\_name: "abc", status: "red", timed\_out: false, number\_of\_nodes: 2, number\_of\_data\_nodes: 2, active\_primary\_shards: 4226, active\_shards: 8452, relocating\_shards: 0, initializing\_shards: 0, unassigned\_shards: 3781, delayed\_unassigned\_shards: 0, number\_of\_pending\_tasks: 0, number\_of\_in\_flight\_fetch: 0 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 27, 2017, 4:12am UTC](https://discuss.elastic.co/t/how-to-recover-a-failed-data-node-in-production-cluster/76547/6 "2017-02-27T04:12:45Z")
> [@ImdotnetJunkie](#): > > number\_of\_nodes: 2,number\_of\_data\_nodes: 2,active\_primary\_shards: 4226,active\_shards: 8452, That's faaaaaaar too many shards and likely to be causing your issues. ---
**Author:** ![ImdotnetJunkie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/imdotnetjunkie/32/15514_2.png) [@ImdotnetJunkie](https://discuss.elastic.co/u/ImdotnetJunkie)\ **Post date:** [February 27, 2017, 4:19am UTC](https://discuss.elastic.co/t/how-to-recover-a-failed-data-node-in-production-cluster/76547/7 "2017-02-27T04:19:40Z")
We have got around 800+ indices. How can this be resolved ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 27, 2017, 4:24am UTC](https://discuss.elastic.co/t/how-to-recover-a-failed-data-node-in-production-cluster/76547/8 "2017-02-27T04:24:27Z")
Reindex. You will need to do that to get to 5.X anyway, so it might make sense to start a 5.X cluster, then run a remote index to pull the data from that old cluster - assuming you still want it. ---
**Author:** ![ImdotnetJunkie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/imdotnetjunkie/32/15514_2.png) [@ImdotnetJunkie](https://discuss.elastic.co/u/ImdotnetJunkie)\ **Post date:** [February 27, 2017, 4:28am UTC](https://discuss.elastic.co/t/how-to-recover-a-failed-data-node-in-production-cluster/76547/9 "2017-02-27T04:28:12Z")
Thanks @warkolm. I will get back if I have any further questions. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 27, 2017, 4:28am UTC](https://discuss.elastic.co/t/how-to-recover-a-failed-data-node-in-production-cluster/76547/10 "2017-03-27T04:28:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.