# Unexpected cluster state **URL:** \ **Category:** Elasticsearch\ **Created:** [August 13, 2013, 7:15pm UTC](https://discuss.elastic.co/t/unexpected-cluster-state/13204 "2013-08-13T19:15:58Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Dave\_Konopka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dave_konopka/32/2235_2.png) [@Dave\_Konopka](https://discuss.elastic.co/u/Dave_Konopka)\ **Post date:** [August 13, 2013, 7:15pm UTC](https://discuss.elastic.co/t/unexpected-cluster-state/13204/1 "2013-08-13T19:15:58Z")
Hello, We're running a three node cluster with the following discovery settings: discovery.zen.minimum\_master\_nodes: 2 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: \ Yesterday we had a networking blip that affected at least one of the nodes. After the networking issue resolved node 1 and 2 were connected to each other and in a green cluster state. Node 3 was connected to node 2 and reporting 2 nodes in the cluster and in a yellow state. Querying the nodes on 1 & 2 showed 1 & 2 were members. On 3 it was reporting 2 & 3 were members. 1 cluster health was reporting unallocated shards. 3 was reporting 200 status for the node. We restarted the service on 3 and it rejoined the cluster properly. Does this scenario sound familiar to anyone? How is it that 1 & 2 and 2 & 3 would join each other separately? Is there any way to avoid this situation? Thanks, Dave -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Oli\_McCormack](https://avatars.discourse-cdn.com/v4/letter/o/fbc32d/32.png) [@Oli\_McCormack](https://discuss.elastic.co/u/Oli_McCormack)\ **Post date:** [August 13, 2013, 7:55pm UTC](https://discuss.elastic.co/t/unexpected-cluster-state/13204/2 "2013-08-13T19:55:51Z")
Hi Dave, I think that you're seeing this issue: [minimum\_master\_nodes does not prevent split-brain if splits are intersecting · Issue #2488 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2488). We were affected by this also. I'm currently trying an alternative approach to the default discovery mechanism, assessing zookeeper and the corresponding plugin[https://github.com/sonian/elasticsearch-zookeeper](https://github.com/sonian/elasticsearch-zookeeper) with our cluster (as suggested in that ticket), which so far has proved successful in avoiding this situation. - oli On Tue, Aug 13, 2013 at 12:15 PM, Dave Konopka [dave.konopka@gmail.com](mailto:dave.konopka@gmail.com)wrote: > Hello, > > We're running a three node cluster with the following discovery settings: > > discovery.zen.minimum\_master\_nodes: 2 > discovery.zen.ping.multicast.enabled: false > discovery.zen.ping.unicast.hosts: \ > > Yesterday we had a networking blip that affected at least one of the > nodes. After the networking issue resolved node 1 and 2 were connected to > each other and in a green cluster state. Node 3 was connected to node 2 and > reporting 2 nodes in the cluster and in a yellow state. Querying the nodes > on 1 & 2 showed 1 & 2 were members. On 3 it was reporting 2 & 3 were > members. 1 cluster health was reporting unallocated shards. 3 was reporting > 200 status for the node. > > We restarted the service on 3 and it rejoined the cluster properly. > > Does this scenario sound familiar to anyone? How is it that 1 & 2 and 2 & > 3 would join each other separately? Is there any way to avoid this > situation? > > Thanks, > Dave > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Dave\_Konopka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dave_konopka/32/2235_2.png) [@Dave\_Konopka](https://discuss.elastic.co/u/Dave_Konopka)\ **Post date:** [August 13, 2013, 8:15pm UTC](https://discuss.elastic.co/t/unexpected-cluster-state/13204/3 "2013-08-13T20:15:24Z")
Oli, Thanks for the pointer. I'll definitely dig into this issue thread. I went through the server logs and they seem to align with the basic premise of that issue description. Node 1 - 3 failed pings to Node 3, removed from the cluster - Few node disconnect exceptions Node 2 - Remove Node 3, told to do so by Node 1 - New master announces itself, Node 3 - Few suspect illegal state warnings - New master announces itself, Node 1 - Master left, Node 3 - Added Node 3, told to do so by Node 1 Node 3 - Master left: Node 1 - New master: Node 3 - Shutdown, startup sequence - Detected master: Node 1 Dave On Tue, Aug 13, 2013 at 3:55 PM, Oli McCormack [oli@climate.com](mailto:oli@climate.com) wrote: > Hi Dave, > > I think that you're seeing this issue: > [minimum\_master\_nodes does not prevent split-brain if splits are intersecting · Issue #2488 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2488). We were > affected by this also. > > I'm currently trying an alternative approach to the default discovery > mechanism, assessing zookeeper and the corresponding plugin[https://github.com/sonian/elasticsearch-zookeeper](https://github.com/sonian/elasticsearch-zookeeper) with > our cluster (as suggested in that ticket), which so far has proved > successful in avoiding this situation. > > - oli > > On Tue, Aug 13, 2013 at 12:15 PM, Dave Konopka [dave.konopka@gmail.com](mailto:dave.konopka@gmail.com)wrote: > > > Hello, > > > > We're running a three node cluster with the following discovery settings: > > > > discovery.zen.minimum\_master\_nodes: 2 > > discovery.zen.ping.multicast.enabled: false > > discovery.zen.ping.unicast.hosts: \ > config\> > > > > Yesterday we had a networking blip that affected at least one of the > > nodes. After the networking issue resolved node 1 and 2 were connected to > > each other and in a green cluster state. Node 3 was connected to node 2 and > > reporting 2 nodes in the cluster and in a yellow state. Querying the nodes > > on 1 & 2 showed 1 & 2 were members. On 3 it was reporting 2 & 3 were > > members. 1 cluster health was reporting unallocated shards. 3 was reporting > > 200 status for the node. > > > > We restarted the service on 3 and it rejoined the cluster properly. > > > > Does this scenario sound familiar to anyone? How is it that 1 & 2 and 2 & > > 3 would join each other separately? Is there any way to avoid this > > situation? > > > > Thanks, > > Dave > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [August 14, 2013, 5:08pm UTC](https://discuss.elastic.co/t/unexpected-cluster-state/13204/4 "2013-08-14T17:08:57Z")
Which version of elasticsearch are you running? I found the logs to not be too helpful when it comes to having some insights into the master election process. One useful tool is the 'lifecycle' command of this script: > **[GitHub - elastic/es2unix: Command-line ES](https://github.com/elastic/es2unix)** > > Command-line ES. Contribute to elastic/es2unix development by creating an account on GitHub. -- Ivan On Tue, Aug 13, 2013 at 1:15 PM, Dave Konopka [dave.konopka@gmail.com](mailto:dave.konopka@gmail.com)wrote: > Oli, > > Thanks for the pointer. I'll definitely dig into this issue thread. > > I went through the server logs and they seem to align with the basic > premise of that issue description. > > Node 1 > > - 3 failed pings to Node 3, removed from the cluster > - Few node disconnect exceptions > > Node 2 > > - Remove Node 3, told to do so by Node 1 > - New master announces itself, Node 3 > - Few suspect illegal state warnings > - New master announces itself, Node 1 > - Master left, Node 3 > - Added Node 3, told to do so by Node 1 > > Node 3 > > - Master left: Node 1 > - New master: Node 3 > - Shutdown, startup sequence > - Detected master: Node 1 > > Dave > > On Tue, Aug 13, 2013 at 3:55 PM, Oli McCormack [oli@climate.com](mailto:oli@climate.com) wrote: > > > Hi Dave, > > > > I think that you're seeing this issue: > > [minimum\_master\_nodes does not prevent split-brain if splits are intersecting · Issue #2488 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2488). We were > > affected by this also. > > > > I'm currently trying an alternative approach to the default discovery > > mechanism, assessing zookeeper and the corresponding plugin[https://github.com/sonian/elasticsearch-zookeeper](https://github.com/sonian/elasticsearch-zookeeper) with > > our cluster (as suggested in that ticket), which so far has proved > > successful in avoiding this situation. > > > > - oli > > > > On Tue, Aug 13, 2013 at 12:15 PM, Dave Konopka [dave.konopka@gmail.com](mailto:dave.konopka@gmail.com)wrote: > > > > > Hello, > > > > > > We're running a three node cluster with the following discovery settings: > > > > > > discovery.zen.minimum\_master\_nodes: 2 > > > discovery.zen.ping.multicast.enabled: false > > > discovery.zen.ping.unicast.hosts: \ > > config\> > > > > > > Yesterday we had a networking blip that affected at least one of the > > > nodes. After the networking issue resolved node 1 and 2 were connected to > > > each other and in a green cluster state. Node 3 was connected to node 2 and > > > reporting 2 nodes in the cluster and in a yellow state. Querying the nodes > > > on 1 & 2 showed 1 & 2 were members. On 3 it was reporting 2 & 3 were > > > members. 1 cluster health was reporting unallocated shards. 3 was reporting > > > 200 status for the node. > > > > > > We restarted the service on 3 and it rejoined the cluster properly. > > > > > > Does this scenario sound familiar to anyone? How is it that 1 & 2 and 2 > > > & 3 would join each other separately? Is there any way to avoid this > > > situation? > > > > > > Thanks, > > > Dave > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Dave\_Konopka](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dave_konopka/32/2235_2.png) [@Dave\_Konopka](https://discuss.elastic.co/u/Dave_Konopka)\ **Post date:** [August 14, 2013, 9:01pm UTC](https://discuss.elastic.co/t/unexpected-cluster-state/13204/5 "2013-08-14T21:01:43Z")
Ivan, We're running 0.90.1. Thanks for pointing out es2unix. This looks handy. Dave On Wed, Aug 14, 2013 at 1:08 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > Which version of elasticsearch are you running? I found the logs to not be > too helpful when it comes to having some insights into the master election > process. > > One useful tool is the 'lifecycle' command of this script: > [GitHub - elastic/es2unix: Command-line ES](https://github.com/elasticsearch/es2unix) > > -- > Ivan > > On Tue, Aug 13, 2013 at 1:15 PM, Dave Konopka [dave.konopka@gmail.com](mailto:dave.konopka@gmail.com)wrote: > > > Oli, > > > > Thanks for the pointer. I'll definitely dig into this issue thread. > > > > I went through the server logs and they seem to align with the basic > > premise of that issue description. > > > > Node 1 > > > > - 3 failed pings to Node 3, removed from the cluster > > - Few node disconnect exceptions > > > > Node 2 > > > > - Remove Node 3, told to do so by Node 1 > > - New master announces itself, Node 3 > > - Few suspect illegal state warnings > > - New master announces itself, Node 1 > > - Master left, Node 3 > > - Added Node 3, told to do so by Node 1 > > > > Node 3 > > > > - Master left: Node 1 > > - New master: Node 3 > > - Shutdown, startup sequence > > - Detected master: Node 1 > > > > Dave > > > > On Tue, Aug 13, 2013 at 3:55 PM, Oli McCormack [oli@climate.com](mailto:oli@climate.com) wrote: > > > > > Hi Dave, > > > > > > I think that you're seeing this issue: > > > [minimum\_master\_nodes does not prevent split-brain if splits are intersecting · Issue #2488 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2488). We were > > > affected by this also. > > > > > > I'm currently trying an alternative approach to the default discovery > > > mechanism, assessing zookeeper and the corresponding plugin[https://github.com/sonian/elasticsearch-zookeeper](https://github.com/sonian/elasticsearch-zookeeper) with > > > our cluster (as suggested in that ticket), which so far has proved > > > successful in avoiding this situation. > > > > > > - oli > > > > > > On Tue, Aug 13, 2013 at 12:15 PM, Dave Konopka [dave.konopka@gmail.com](mailto:dave.konopka@gmail.com)wrote: > > > > > > > Hello, > > > > > > > > We're running a three node cluster with the following discovery > > > > settings: > > > > > > > > discovery.zen.minimum\_master\_nodes: 2 > > > > discovery.zen.ping.multicast.enabled: false > > > > discovery.zen.ping.unicast.hosts: \ > > > config\> > > > > > > > > Yesterday we had a networking blip that affected at least one of the > > > > nodes. After the networking issue resolved node 1 and 2 were connected to > > > > each other and in a green cluster state. Node 3 was connected to node 2 and > > > > reporting 2 nodes in the cluster and in a yellow state. Querying the nodes > > > > on 1 & 2 showed 1 & 2 were members. On 3 it was reporting 2 & 3 were > > > > members. 1 cluster health was reporting unallocated shards. 3 was reporting > > > > 200 status for the node. > > > > > > > > We restarted the service on 3 and it rejoined the cluster properly. > > > > > > > > Does this scenario sound familiar to anyone? How is it that 1 & 2 and 2 > > > > & 3 would join each other separately? Is there any way to avoid this > > > > situation? > > > > > > > > Thanks, > > > > Dave > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:21am UTC](https://discuss.elastic.co/t/unexpected-cluster-state/13204/6 "2017-07-06T02:21:06Z")