# Cluster Recovery took long time

**URL:** https://discuss.elastic.co/t/cluster-recovery-took-long-time/143489
**Category:** Elasticsearch
**Created:** [August 8, 2018, 10:24am UTC](https://discuss.elastic.co/t/cluster-recovery-took-long-time/143489 "2018-08-08T10:24:50Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![mahmoud\_moharam](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mahmoud\_moharam](https://discuss.elastic.co/u/mahmoud_moharam)
#### Post date: [August 8, 2018, 10:24am UTC](https://discuss.elastic.co/t/cluster-recovery-took-long-time/143489/1 "2018-08-08T10:24:50Z")

</div>

I have a cluster with 2 nodes on AWS , then I release one of them and keep another so my cluster now had 1 master node , with active\_primary\_shards 38180  
the problem is when I test failover time recovery it took **12 hours** to recover and cluster status turned into "yellow" ![jp](https://us1.discourse-cdn.com/elastic/original/3X/8/9/899bf68b2cad8264633797c6046d3f0927646428.jpg) as shown in attached image

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 8, 2018, 11:05am UTC](https://discuss.elastic.co/t/cluster-recovery-took-long-time/143489/2 "2018-08-08T11:05:48Z")

</div>

You have far too many shards for a cluster that size. Read [this blog post for some guidance on shards and sharding practices](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster).

---

<div class="post-metadata">

### Author: ![mahmoud\_moharam](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mahmoud\_moharam](https://discuss.elastic.co/u/mahmoud_moharam)
#### Post date: [August 8, 2018, 11:27am UTC](https://discuss.elastic.co/t/cluster-recovery-took-long-time/143489/3 "2018-08-08T11:27:45Z")

</div>

thanks for reply @Christian_Dahlqvist , so now I know that I had too many shards , is there any solution to avoid this deadly time when recover or even a solution to re-shard the cluster ??

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 8, 2018, 11:31am UTC](https://discuss.elastic.co/t/cluster-recovery-took-long-time/143489/4 "2018-08-08T11:31:39Z")

</div>

With that number of shards I am surprised your cluster is running at all. I think it may be the largest shard count I have ever seen for a single node. As you only have 1 node your replicas will never allocate, so it seems fine, at least from that perspective.

---

<div class="post-metadata">

### Author: ![mahmoud\_moharam](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mahmoud\_moharam](https://discuss.elastic.co/u/mahmoud_moharam)
#### Post date: [August 8, 2018, 11:47am UTC](https://discuss.elastic.co/t/cluster-recovery-took-long-time/143489/5 "2018-08-08T11:47:21Z")

</div>

so , is there anyway to avoid this single point of failure , to avoid cluster failure and took 12 hours to recover , such as adding new nodes ?

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 8, 2018, 12:53pm UTC](https://discuss.elastic.co/t/cluster-recovery-took-long-time/143489/6 "2018-08-08T12:53:19Z")

</div>

Adding nodes may help and provide some temporary relief, but I am quite sure you will still need to change your sharding practices.

---

<div class="post-metadata">

### Author: ![mahmoud\_moharam](https://avatars.discourse-cdn.com/v4/letter/m/96bed5/32.png) [@mahmoud\_moharam](https://discuss.elastic.co/u/mahmoud_moharam)
#### Post date: [August 8, 2018, 12:56pm UTC](https://discuss.elastic.co/t/cluster-recovery-took-long-time/143489/7 "2018-08-08T12:56:05Z")

</div>

ok thanks for your kind help , so If I decide to sharding again what is the best options ?

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [August 8, 2018, 12:57pm UTC](https://discuss.elastic.co/t/cluster-recovery-took-long-time/143489/8 "2018-08-08T12:57:37Z")

</div>

As outlined in the blog I linked to, try to make sure you have quite large shards as having lots of small indices and shards is very inefficient.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 5, 2018, 12:57pm UTC](https://discuss.elastic.co/t/cluster-recovery-took-long-time/143489/9 "2018-09-05T12:57:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
