[2020-01-06T06:26:46,981][WARN ][o.e.c.c.ClusterFormationFailureHelper] [Master-node] master not discovered or elected yet, an election requires a node with id [Y11KbKB-SX2HIYM77Z1Zqg], have discovered [{Master-node}{PcrrIC01SECgC-gDmABeSA}{sSuci3QXRSSN4JyZ1XaijQ}{ServerA}{ip_address of server A:9300}{ilm}{ml.machine_memory=51539005440, xpack.installed=true, ml.max_open_jobs=20}] which is not a quorum; discovery will continue using [ip_address of server B:9300, ip_address of server C:9300] from hosts providers and [{Master-node}{PcrrIC01SECgC-gDmABeSA}{sSuci3QXRSSN4JyZ1XaijQ}{ServerA}{ip_address of server A:9300}{ilm}{ml.machine_memory=51539005440, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 66, last-accepted version 1011 in term 66
So Master-node is looking in vain for a node with ID Y11KbKB-SX2HIYM77Z1Zqg. It is not discovering any other nodes. I suspect that Y11KbKB-SX2HIYM77Z1Zqg is the node ID of Data-node1. Check the first few log lines on Data-node1 to find its node ID to confirm or otherwise.
I recommend fixing the port of your master-eligible nodes using transport.tcp.port: 9300. This means both Master-node and Data-node1. You are currently using 9300-9400 but if Elasticsearch picks a port other than 9300 then you need to configure discovery.zen.ping.unicast.hosts with that port instead.
Note also that in v7 you should replace discovery.zen.ping.unicast.hosts with discovery.seed_hosts, and discovery.zen.minimum_master_nodes should be removed.
Note also that there's no point in having two master-eligible nodes. I recommend setting node.master: true on all three nodes.
ID Y11KbKB-SX2HIYM77Z1Zqg is of data node 3. I have checked the logs for the same.
I have changed transport.tcp.port: 9300-9400 settings to only 9300 port.
I have also repplaced discovery.zen.ping.unicast.hosts with discovery.seed_hosts
I have also changed the setting for node.master: true for all the 3 nodes.
I am getting below message after run of all the 3 nodes on master node:
[2020-01-06T12:45:41,576][INFO ][o.e.x.m.e.l.LocalExporter] [Master-node] waiting for elected master node [{Data-node2}{Y11KbKB-SX2HIYM77Z1Zqg}{UrZFy5cOQ7Ck3dBs3HEH0Q}{server C}{ip_address of server B:9300}{dlm}{ml.machine_memory=17179267072, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter[default_local] (does it have x-pack installed?)
[2020-01-06T12:45:41,826][INFO ][o.e.c.s.ClusterSettings ] [Master-node] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2020-01-06T12:45:41,998][INFO ][o.e.l.LicenseService ] [Master-node] license [abb78209-b323-4d4a-8782-a8f7eaf744d4] mode [basic] - valid
[2020-01-06T12:45:42,014][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [Master-node] Active license is now [BASIC]; Security is disabled
[2020-01-06T12:45:42,295][INFO ][o.e.x.s.a.TokenService ] [Master-node] refresh keys
[2020-01-06T12:45:42,529][INFO ][o.e.x.s.a.TokenService ] [Master-node] refreshed keys
Is this correct? means can i say now ES cluster is up and running?
Thanks for update.
I have one more question.
All 3 nodes are master eligible. So if one goes down other should be eligible to elect itself as a master node and will form a 2 node cluster.
How i can check this with my cluster?
Can you please guide me on this more?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.