# Master node can not be elected After old master process got killed **URL:** \ **Category:** Elasticsearch\ **Created:** [May 13, 2013, 7:01am UTC](https://discuss.elastic.co/t/master-node-can-not-be-elected-after-old-master-process-got-killed/11929 "2013-05-13T07:01:32Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![xiaoliang\_tian](https://avatars.discourse-cdn.com/v4/letter/x/a9a28c/32.png) [@xiaoliang\_tian](https://discuss.elastic.co/u/xiaoliang_tian)\ **Post date:** [May 13, 2013, 7:01am UTC](https://discuss.elastic.co/t/master-node-can-not-be-elected-after-old-master-process-got-killed/11929/1 "2013-05-13T07:01:32Z")
Here is my first configuration: discovery.zen.minimum\_master\_nodes: 1 discovery.zen.ping.unicast.hosts: [10.224.2.159,10.152.150.57,10.147.197.172,10.166.65.202] If I kill master node ,The whole cluster is totally a mess.It doesn't elect new master for the left 3 nodes \* \* _if query message,there is exception "BroadcastShardOperationFailedException" in node 2 and node 3.node 1 and node 4 is fine_ Here is my second configuration: discovery.zen.minimum\_master\_nodes: 4 discovery.zen.ping.unicast.hosts: [10.224.2.159,10.152.150.57,10.147.197.172,10.166.65.202] If I kill master node ,The whole cluster hang \* - there is no any information about cluster or nodes in bigdesk web console , besides check "\_cluster/health" return "{"error":"MasterNotDiscoveredException[waited for [30s]]","status":503}" - - So I assume ,If I give 4 IP address to "discovery.zen.ping.unicast.hosts: " All of this 4 IP can not leave this cluster,Otherwise the cluster will not work properly Am I right? My Cluster is running on EC2 .How should I do to make Those nodes elect new master If The old master dead? Thanks \* -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [May 14, 2013, 10:34pm UTC](https://discuss.elastic.co/t/master-node-can-not-be-elected-after-old-master-process-got-killed/11929/2 "2013-05-14T22:34:14Z")
田晓亮 , Could you be missing the _configuration item_ that disables multicast? (The first in the 3 configuration items below.) Here are the configuration items I used for a 3-node cluster using unicast discovery and 2 masters required. The host names are changed but the basic configuration should be easily understood: _discovery.zen.ping.multicast.enabled=false_ discovery.zen.ping.unicast.hosts=host1,host2,host3 discovery.zen.minimum\_master\_nodes=2 Also, the largest index was configured with 2 replicas, so there exists one copy of a shard on each node. And note that a minimum of 2 masters is defined, but the entire cluster is configured as 3 nodes (could be 4 if I had another server). This cluster successfully surived a thermal shutdown of the third host. The cluster state remained green with the first two hosts handling updates and queries. When the third host was brought up, there were a few minutes where the largest index (97M records across 5 shards) took some time to re-assign / re-verify the shards to the newly restored host. And now the cluster is back in operation. Never missed a beat. Hope this helps! Regards, Brian On Monday, May 13, 2013 3:01:32 AM UTC-4, 田晓亮 wrote: > Here is my first configuration: > > discovery.zen.minimum\_master\_nodes: 1 > > discovery.zen.ping.unicast.hosts: > [10.224.2.159,10.152.150.57,10.147.197.172,10.166.65.202] > > If I kill master node ,The whole cluster is totally a mess.It doesn't > elect new master for the left 3 nodes > \* > \* > _if query message,there is exception > "BroadcastShardOperationFailedException" in node 2 and node 3.node 1 and > node 4 is fine_ > > Here is my second configuration: > > discovery.zen.minimum\_master\_nodes: 4 > > discovery.zen.ping.unicast.hosts: > [10.224.2.159,10.152.150.57,10.147.197.172,10.166.65.202] > > If I kill master node ,The whole cluster hang > \* > > - > > there is no any information about cluster or nodes in bigdesk web > console , besides check "\_cluster/health" return > "{"error":"MasterNotDiscoveredException[waited for [30s]]","status":503}" > > - > - > > So I assume ,If I give 4 IP address to "discovery.zen.ping.unicast.hosts: > " > > All of this 4 IP can not leave this cluster,Otherwise the cluster will not > work properly > > Am I right? > > My Cluster is running on EC2 .How should I do to make Those nodes elect > new master If The old master dead? > > Thanks > \* -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![xiaoliang\_tian](https://avatars.discourse-cdn.com/v4/letter/x/a9a28c/32.png) [@xiaoliang\_tian](https://discuss.elastic.co/u/xiaoliang_tian)\ **Post date:** [May 15, 2013, 10:17am UTC](https://discuss.elastic.co/t/master-node-can-not-be-elected-after-old-master-process-got-killed/11929/3 "2013-05-15T10:17:16Z")
That really helps a lot,Thanks 2013/5/15 InquiringMind [brian.from.fl@gmail.com](mailto:brian.from.fl@gmail.com) > 田晓亮 , > > Could you be missing the _configuration item_ that disables multicast? > (The first in the 3 configuration items below.) > > Here are the configuration items I used for a 3-node cluster using unicast > discovery and 2 masters required. The host names are changed but the basic > configuration should be easily understood: > > _discovery.zen.ping.multicast.enabled=false_ > discovery.zen.ping.unicast.hosts=host1,host2,host3 > discovery.zen.minimum\_master\_nodes=2 > > Also, the largest index was configured with 2 replicas, so there exists > one copy of a shard on each node. > > And note that a minimum of 2 masters is defined, but the entire cluster is > configured as 3 nodes (could be 4 if I had another server). > > This cluster successfully surived a thermal shutdown of the third host. > The cluster state remained green with the first two hosts handling updates > and queries. When the third host was brought up, there were a few minutes > where the largest index (97M records across 5 shards) took some time to > re-assign / re-verify the shards to the newly restored host. And now the > cluster is back in operation. Never missed a beat. > > Hope this helps! > > Regards, > Brian > > On Monday, May 13, 2013 3:01:32 AM UTC-4, 田晓亮 wrote: > > > Here is my first configuration: > > > > discovery.zen.minimum\_master\_\*\*nodes: 1 > > > > discovery.zen.ping.unicast.**hosts: [10.224.2.159,10.152.150.57,** > > 10.147.197.172,10.166.65.202] > > > > If I kill master node ,The whole cluster is totally a mess.It doesn't > > elect new master for the left 3 nodes > > \* > > \* > > _if query message,there is exception "BroadcastShardOperationFailedException" > > in node 2 and node 3.node 1 and node 4 is fine_ > > > > Here is my second configuration: > > > > discovery.zen.minimum\_master\_\*\*nodes: 4 > > > > discovery.zen.ping.unicast.**hosts: [10.224.2.159,10.152.150.57,** > > 10.147.197.172,10.166.65.202] > > > > If I kill master node ,The whole cluster hang > > \* > > > > - > > > > there is no any information about cluster or nodes in bigdesk web > > console , besides check "\_cluster/health" return "{"error":" > > MasterNotDiscoveredException[waited for [30s]]","status":503}" > > > > - > > - > > > > So I assume ,If I give 4 IP address to "discovery.zen.ping.unicast. > > hosts: > > " > > > > All of this 4 IP can not leave this cluster,Otherwise the cluster will > > not work properly > > > > Am I right? > > > > My Cluster is running on EC2 .How should I do to make Those nodes elect > > new master If The old master dead? > > > > Thanks > > \* > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/FQ3KwTKy3r0/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/FQ3KwTKy3r0/unsubscribe?hl=en-US) > > . > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:36am UTC](https://discuss.elastic.co/t/master-node-can-not-be-elected-after-old-master-process-got-killed/11929/4 "2017-07-06T02:36:38Z")