# Stress Free Guide To Expanding a Cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [June 25, 2014, 12:05pm UTC](https://discuss.elastic.co/t/stress-free-guide-to-expanding-a-cluster/18329 "2014-06-25T12:05:30Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![James\_Carr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_carr/32/956_2.png) [@James\_Carr](https://discuss.elastic.co/u/James_Carr)\ **Post date:** [June 25, 2014, 12:05pm UTC](https://discuss.elastic.co/t/stress-free-guide-to-expanding-a-cluster/18329/1 "2014-06-25T12:05:30Z")
Earlier this week we discovered that our three node elasticsearch cluster needed to be expanded as it was getting dangerously close to maximum capacity. I was nervous about this and read up the best I could on best practices to doing this. The only information I seemed to be able to find is to ensure that the new nodes cannot be elected as masters when they join to avoid a split brain scenario. Fair enough. I launched two new EC2 instances to join the cluster and watched. Some shards began relocating, no big deal. Six hours later I checked in and some shards were still locating, one shard was recovering. Weird but whatever... the cluster health is still green and searches are working fine. Then I got an alert at 2:30am that the cluster state is now yellow and find that we have 3 shards marked as recovering and 2 shards that unassigned. The cluster still technically works but 24 hours later after the new nodes were added I feel like my only choice to get a green cluster again will be to simply launch 5 fresh nodes and replay all the data from backups into it. Ugggggh. SERIOUSLY! What can I do to prevent this? I feel like I am missing something because I always heard the strength of elasticsearch is its ease of scaling out but it feels like every time I try it falls to the floor. ☹ Thanks! James -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJreXKD3Wuyiq5XxGdSWyj3a%3DM2Xd5GQxZ9J3EywoT-OP52qFA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJreXKD3Wuyiq5XxGdSWyj3a%3DM2Xd5GQxZ9J3EywoT-OP52qFA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [June 25, 2014, 12:19pm UTC](https://discuss.elastic.co/t/stress-free-guide-to-expanding-a-cluster/18329/2 "2014-06-25T12:19:35Z")
On Wed, Jun 25, 2014 at 8:05 AM, James Carr [james.r.carr@gmail.com](mailto:james.r.carr@gmail.com) wrote: > I launched two new EC2 instances to join the cluster and watched. Some > shards began relocating, no big deal. Six hours later I checked in and > some shards were still locating, one shard was recovering. Weird but > whatever... the cluster health is still green and searches are working > fine. I add new nodes every once in a while and it can take a few hours for everything to balance out, but six hours is a bit long. Its possible. Do you have graphs of the count of relocating shards? Something like this can really help you figure out if everything balanced out at some point and then unbalanced. Example: [http://ganglia.wikimedia.org/latest/graph\_all\_periods.php?c=Elasticsearch%20cluster%20eqiad&h=elastic1001.eqiad.wmnet&r=hour&z=default&jr=&js=&st=1403698335&v=0&m=es\_relocating\_shards&vl=shards&ti=es\_relocating\_shards&z=large](http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Elasticsearch%20cluster%20eqiad&h=elastic1001.eqiad.wmnet&r=hour&z=default&jr=&js=&st=1403698335&v=0&m=es_relocating_shards&vl=shards&ti=es_relocating_shards&z=large) Then I got an alert at 2:30am that the cluster state is now > yellow and find that we have 3 shards marked as recovering and 2 > shards that unassigned. The cluster still technically works but 24 > hours later after the new nodes were added I feel like my only choice > to get a green cluster again will be to simply launch 5 fresh nodes > and replay all the data from backups into it. Ugggggh. This sounds like one of the nodes bounced. It can take a long time to recover from that. Its something that is being worked on. Check the logs and see if you see anything about it. One thing to make sure of is that you set the number of master nodes correctly on all nodes. If you have five master eligible nodes then set it to 3. If the two new nodes aren't master eligible (you have three master eligible nodes) then set it to 2. > SERIOUSLY! What can I do to prevent this? I feel like I am missing > something because I always heard the strength of elasticsearch is its > ease of scaling out but it feels like every time I try it falls to the > floor. ☹ Its always been pretty painless for me. I did have trouble when I added nodes that were broken: one time I added nodes without SSDs to a cluster with SSDs. Another time I didn't set the heap size on the new nodes and they worked until some shards moved to them. Then they fell over. Nik -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0CNzBEv6HC8J-P91qHS46Micb7VjmO2LTXN4JY2QGCkg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0CNzBEv6HC8J-P91qHS46Micb7VjmO2LTXN4JY2QGCkg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Michael\_Hart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael_hart/32/65937_2.png) [@Michael\_Hart](https://discuss.elastic.co/u/Michael_Hart)\ **Post date:** [June 25, 2014, 3:28pm UTC](https://discuss.elastic.co/t/stress-free-guide-to-expanding-a-cluster/18329/3 "2014-06-25T15:28:38Z")
Try setting "indices.recovery.max\_bytes\_per\_sec" much higher for faster recovery. The default is 20mb/s, and there's a bug in versions prior to 1.2 that rate limit to even lower than that. You didn't specify how big your indices are, but I can fairly accurately predict how long it'll take for the cluster to go green with that parameter. mike On Wednesday, June 25, 2014 8:20:02 AM UTC-4, Nikolas Everett wrote: > On Wed, Jun 25, 2014 at 8:05 AM, James Carr \<[james....@gmail.com](mailto:james....@gmail.com) > \\> wrote: > > > I launched two new EC2 instances to join the cluster and watched. Some > > shards began relocating, no big deal. Six hours later I checked in and > > some shards were still locating, one shard was recovering. Weird but > > whatever... the cluster health is still green and searches are working > > fine. > > I add new nodes every once in a while and it can take a few hours for > everything to balance out, but six hours is a bit long. Its possible. Do > you have graphs of the count of relocating shards? Something like this can > really help you figure out if everything balanced out at some point and > then unbalanced. Example: > [http://ganglia.wikimedia.org/latest/graph\_all\_periods.php?c=Elasticsearch%20cluster%20eqiad&h=elastic1001.eqiad.wmnet&r=hour&z=default&jr=&js=&st=1403698335&v=0&m=es\_relocating\_shards&vl=shards&ti=es\_relocating\_shards&z=large](http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Elasticsearch%20cluster%20eqiad&h=elastic1001.eqiad.wmnet&r=hour&z=default&jr=&js=&st=1403698335&v=0&m=es_relocating_shards&vl=shards&ti=es_relocating_shards&z=large) > > Then I got an alert at 2:30am that the cluster state is now > > > yellow and find that we have 3 shards marked as recovering and 2 > > shards that unassigned. The cluster still technically works but 24 > > hours later after the new nodes were added I feel like my only choice > > to get a green cluster again will be to simply launch 5 fresh nodes > > and replay all the data from backups into it. Ugggggh. > > This sounds like one of the nodes bounced. It can take a long time to > recover from that. Its something that is being worked on. Check the logs > and see if you see anything about it. > > One thing to make sure of is that you set the number of master nodes > correctly on all nodes. If you have five master eligible nodes then set it > to 3. If the two new nodes aren't master eligible (you have three master > eligible nodes) then set it to 2. > > > SERIOUSLY! What can I do to prevent this? I feel like I am missing > > something because I always heard the strength of elasticsearch is its > > ease of scaling out but it feels like every time I try it falls to the > > floor. ☹ > > Its always been pretty painless for me. I did have trouble when I added > nodes that were broken: one time I added nodes without SSDs to a cluster > with SSDs. Another time I didn't set the heap size on the new nodes and > they worked until some shards moved to them. Then they fell over. > > Nik -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/17a60021-e0bc-4806-8573-f37a9ef91b89%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/17a60021-e0bc-4806-8573-f37a9ef91b89%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:19am UTC](https://discuss.elastic.co/t/stress-free-guide-to-expanding-a-cluster/18329/4 "2017-07-06T01:19:46Z")