# Cluster troubles, Azure related? **URL:** \ **Category:** Elasticsearch\ **Created:** [September 16, 2014, 6:00pm UTC](https://discuss.elastic.co/t/cluster-troubles-azure-related/19821 "2014-09-16T18:00:33Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Tim\_Heikell](https://avatars.discourse-cdn.com/v4/letter/t/f4b2a3/32.png) [@Tim\_Heikell](https://discuss.elastic.co/u/Tim_Heikell)\ **Post date:** [September 16, 2014, 6:00pm UTC](https://discuss.elastic.co/t/cluster-troubles-azure-related/19821/1 "2014-09-16T18:00:33Z")
We are prepping to launch our app into production and seem to be having some stability issues. We have a cluster of 4 VMs on Azure that all use the Azure plugin for discovery. Most of the time it works as expected, but sometimes it looses its mind. This morning for example, I made adjustments to the memory allocated to the JVM of all nodes. I rebooted all of the nodes, one at a time, waiting for a green status before rebooting the next node. When I rebooted the fourth node, the cluster status turned red (as per node #1). Node 1 only reported that nodes 1 and 2 were in the cluster. I waited and nothing changed. I eventually checked the node status on node 3 and found that nodes 3 and 4 had formed their own cluster. I ended up in a state where nodes 1 and 2 were in a cluster, with 2 being the master, while 3 and 4 were in a separate cluster, with 3 being the master. I stopped the elasticsearch service on 3 and 4 and then started the services up again. They correctly found the cluster of nodes 1 and 2 and all is well again. Why would this happen, and how can I prevent it from happening? On node three I found some interesting log reports that I have copied to > Thanks. Tim -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 16, 2014, 6:21pm UTC](https://discuss.elastic.co/t/cluster-troubles-azure-related/19821/2 "2014-09-16T18:21:06Z")
It looks like you did not configure minimum\_master\_nodes Jörg On Tue, Sep 16, 2014 at 8:00 PM, Tim Heikell [tim.heikell@heapsylon.com](mailto:tim.heikell@heapsylon.com) wrote: > We are prepping to launch our app into production and seem to be having > some stability issues. We have a cluster of 4 VMs on Azure that all use the > Azure plugin for discovery. Most of the time it works as expected, but > sometimes it looses its mind. This morning for example, I made adjustments > to the memory allocated to the JVM of all nodes. I rebooted all of the > nodes, one at a time, waiting for a green status before rebooting the next > node. When I rebooted the fourth node, the cluster status turned red (as > per node #1). Node 1 only reported that nodes 1 and 2 were in the cluster. > I waited and nothing changed. I eventually checked the node status on node > 3 and found that nodes 3 and 4 had formed their own cluster. I ended up in > a state where nodes 1 and 2 were in a cluster, with 2 being the master, > while 3 and 4 were in a separate cluster, with 3 being the master. I > stopped the elasticsearch service on 3 and 4 and then started the services > up again. They correctly found the cluster of nodes 1 and 2 and all is well > again. Why would this happen, and how can I prevent it from happening? On > node three I found some interesting log reports that I have copied to > [gist:9948b1d318cdc4cd0ecf · GitHub](https://gist.github.com/theikell/9948b1d318cdc4cd0ecf) > > Thanks. > > Tim > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGz8b-W%3DpUk1P1Emuszu%3DrnX5%2BwHca7k9he2B59mogoJg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGz8b-W%3DpUk1P1Emuszu%3DrnX5%2BwHca7k9he2B59mogoJg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tim\_Heikell](https://avatars.discourse-cdn.com/v4/letter/t/f4b2a3/32.png) [@Tim\_Heikell](https://discuss.elastic.co/u/Tim_Heikell)\ **Post date:** [September 16, 2014, 6:30pm UTC](https://discuss.elastic.co/t/cluster-troubles-azure-related/19821/3 "2014-09-16T18:30:38Z")
Thanks for the reply Jörg. I have discovery.zen.minimum\_master\_nodes=2. Should it be something different? On Tuesday, September 16, 2014 11:21:16 AM UTC-7, Jörg Prante wrote: > It looks like you did not configure minimum\_master\_nodes > > Jörg > > On Tue, Sep 16, 2014 at 8:00 PM, Tim Heikell \<[tim.h...@heapsylon.com](mailto:tim.h...@heapsylon.com) > \\> wrote: > > > We are prepping to launch our app into production and seem to be having > > some stability issues. We have a cluster of 4 VMs on Azure that all use the > > Azure plugin for discovery. Most of the time it works as expected, but > > sometimes it looses its mind. This morning for example, I made adjustments > > to the memory allocated to the JVM of all nodes. I rebooted all of the > > nodes, one at a time, waiting for a green status before rebooting the next > > node. When I rebooted the fourth node, the cluster status turned red (as > > per node #1). Node 1 only reported that nodes 1 and 2 were in the cluster. > > I waited and nothing changed. I eventually checked the node status on node > > 3 and found that nodes 3 and 4 had formed their own cluster. I ended up in > > a state where nodes 1 and 2 were in a cluster, with 2 being the master, > > while 3 and 4 were in a separate cluster, with 3 being the master. I > > stopped the elasticsearch service on 3 and 4 and then started the services > > up again. They correctly found the cluster of nodes 1 and 2 and all is well > > again. Why would this happen, and how can I prevent it from happening? On > > node three I found some interesting log reports that I have copied to > > [gist:9948b1d318cdc4cd0ecf · GitHub](https://gist.github.com/theikell/9948b1d318cdc4cd0ecf) > > > > Thanks. > > > > Tim > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/53bc6e0c-3110-4f64-90f9-ff0ac84c5ad0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/53bc6e0c-3110-4f64-90f9-ff0ac84c5ad0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Tim\_Heikell](https://avatars.discourse-cdn.com/v4/letter/t/f4b2a3/32.png) [@Tim\_Heikell](https://discuss.elastic.co/u/Tim_Heikell)\ **Post date:** [September 16, 2014, 6:34pm UTC](https://discuss.elastic.co/t/cluster-troubles-azure-related/19821/4 "2014-09-16T18:34:20Z")
Ah, I just found the n/2+1 recommendation, so I expect I need to set it to 3. On Tuesday, September 16, 2014 11:30:38 AM UTC-7, Tim Heikell wrote: > Thanks for the reply Jörg. I have discovery.zen.minimum\_master\_nodes=2. > Should it be something different? > > On Tuesday, September 16, 2014 11:21:16 AM UTC-7, Jörg Prante wrote: > > > It looks like you did not configure minimum\_master\_nodes > > > > Jörg > > > > On Tue, Sep 16, 2014 at 8:00 PM, Tim Heikell [tim.h...@heapsylon.com](mailto:tim.h...@heapsylon.com) > > wrote: > > > > > We are prepping to launch our app into production and seem to be having > > > some stability issues. We have a cluster of 4 VMs on Azure that all use the > > > Azure plugin for discovery. Most of the time it works as expected, but > > > sometimes it looses its mind. This morning for example, I made adjustments > > > to the memory allocated to the JVM of all nodes. I rebooted all of the > > > nodes, one at a time, waiting for a green status before rebooting the next > > > node. When I rebooted the fourth node, the cluster status turned red (as > > > per node #1). Node 1 only reported that nodes 1 and 2 were in the cluster. > > > I waited and nothing changed. I eventually checked the node status on node > > > 3 and found that nodes 3 and 4 had formed their own cluster. I ended up in > > > a state where nodes 1 and 2 were in a cluster, with 2 being the master, > > > while 3 and 4 were in a separate cluster, with 3 being the master. I > > > stopped the elasticsearch service on 3 and 4 and then started the services > > > up again. They correctly found the cluster of nodes 1 and 2 and all is well > > > again. Why would this happen, and how can I prevent it from happening? On > > > node three I found some interesting log reports that I have copied to > > > [gist:9948b1d318cdc4cd0ecf · GitHub](https://gist.github.com/theikell/9948b1d318cdc4cd0ecf) > > > > > > Thanks. > > > > > > Tim > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d2c6462d-8789-4b9f-9776-ea368f7f5661%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9c75e83d-008a-4b05-a62b-23e5e54632d8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9c75e83d-008a-4b05-a62b-23e5e54632d8%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:01am UTC](https://discuss.elastic.co/t/cluster-troubles-azure-related/19821/5 "2017-07-06T01:01:52Z")