o.e.c.c.ClusterFormationFailureHelper] [Master-node] master not discovered or elected yet, an election requires a node with id [Y11KbKB-SX2HIYM77Z1Zqg], have discovered

Hello,

I am trying to form a 3 node ES cluster but i am getting a above mentioned error.
I have one master node and 2 data nodes.

My ES configurations are as follows:

    **For master node:**
    cluster.name: InTouch
    cluster.remote.connect: false
    network.host: ServerA
    transport.host: ServerA
    transport.tcp.port: 9300-9400
    node.name: Master-node
    node.master: true
    node.data: false
    node.ingest: true
    discovery.zen.ping.unicast.hosts: [ServerA, ServerB,ServerC]
    cluster.initial_master_nodes: ["Master-node","Data-node1"]
    discovery.zen.minimum_master_nodes: 2
    bootstrap.memory_lock: true
    network.bind_host:  *local*
    path.data: /elasticsearch/data/
    path.logs: /var/log/elasticsearch/
    path.repo: /elasticsearch/repo/

**For Data node 1**:
cluster.name: InTouch
node.name: Data-node1
node.master: true
node.data: true
node.ingest: false
network.host: ServerB
discovery.zen.ping.unicast.hosts: [ServerA, ServerB,ServerC]
transport.tcp.port: 9300-9400
path.data: /elasticsearch/data/
path.logs: /var/log/elasticsearch/
path.repo: /elasticsearch/repo/

**For Data node 2**:
cluster.name: InTouch
node.name: Data-node2
node.master: false
node.data: true
network.host: ServerC
discovery.zen.ping.unicast.hosts: [ServerA, ServerB,ServerC]
transport.tcp.port: 9300-9400
path.data: /elasticsearch/data/
path.logs: /var/log/elasticsearch/
path.repo: /elasticsearch/repo/

Thanks,
Priyanka

You've not shared the whole log message, and the missing bit is important. Please share the full message.

Hello @DavidTurner,

Please find below error log:

[2020-01-06T06:26:46,981][WARN ][o.e.c.c.ClusterFormationFailureHelper] [Master-node] master not discovered or elected yet, an election requires a node with id [Y11KbKB-SX2HIYM77Z1Zqg], have discovered [{Master-node}{PcrrIC01SECgC-gDmABeSA}{sSuci3QXRSSN4JyZ1XaijQ}{ServerA}{ip_address of server A:9300}{ilm}{ml.machine_memory=51539005440, xpack.installed=true, ml.max_open_jobs=20}] which is not a quorum; discovery will continue using [ip_address of server B:9300, ip_address of server C:9300] from hosts providers and [{Master-node}{PcrrIC01SECgC-gDmABeSA}{sSuci3QXRSSN4JyZ1XaijQ}{ServerA}{ip_address of server A:9300}{ilm}{ml.machine_memory=51539005440, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 66, last-accepted version 1011 in term 66

Regards,
Priyanka

So Master-node is looking in vain for a node with ID Y11KbKB-SX2HIYM77Z1Zqg. It is not discovering any other nodes. I suspect that Y11KbKB-SX2HIYM77Z1Zqg is the node ID of Data-node1. Check the first few log lines on Data-node1 to find its node ID to confirm or otherwise.

I recommend fixing the port of your master-eligible nodes using transport.tcp.port: 9300. This means both Master-node and Data-node1. You are currently using 9300-9400 but if Elasticsearch picks a port other than 9300 then you need to configure discovery.zen.ping.unicast.hosts with that port instead.

Note also that in v7 you should replace discovery.zen.ping.unicast.hosts with discovery.seed_hosts, and discovery.zen.minimum_master_nodes should be removed.

Note also that there's no point in having two master-eligible nodes. I recommend setting node.master: true on all three nodes.

Hello @DavidTurner,

ID Y11KbKB-SX2HIYM77Z1Zqg is of data node 3. I have checked the logs for the same.

I have changed transport.tcp.port: 9300-9400 settings to only 9300 port.

I have also repplaced discovery.zen.ping.unicast.hosts with discovery.seed_hosts

I have also changed the setting for node.master: true for all the 3 nodes.

I am getting below message after run of all the 3 nodes on master node:

 [2020-01-06T12:45:41,576][INFO ][o.e.x.m.e.l.LocalExporter] [Master-node] waiting for elected master node [{Data-node2}{Y11KbKB-SX2HIYM77Z1Zqg}{UrZFy5cOQ7Ck3dBs3HEH0Q}{server C}{ip_address of server B:9300}{dlm}{ml.machine_memory=17179267072, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter[default_local] (does it have x-pack installed?)
[2020-01-06T12:45:41,826][INFO ][o.e.c.s.ClusterSettings  ] [Master-node] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2020-01-06T12:45:41,998][INFO ][o.e.l.LicenseService     ] [Master-node] license [abb78209-b323-4d4a-8782-a8f7eaf744d4] mode [basic] - valid
[2020-01-06T12:45:42,014][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [Master-node] Active license is now [BASIC]; Security is disabled
[2020-01-06T12:45:42,295][INFO ][o.e.x.s.a.TokenService   ] [Master-node] refresh keys
[2020-01-06T12:45:42,529][INFO ][o.e.x.s.a.TokenService   ] [Master-node] refreshed keys

Is this correct? means can i say now ES cluster is up and running?

Regards,
Priyanka

Yes that looks better.

Hello @DavidTurner,

Thanks for update.
I have one more question.
All 3 nodes are master eligible. So if one goes down other should be eligible to elect itself as a master node and will form a 2 node cluster.
How i can check this with my cluster?
Can you please guide me on this more?

Regards,
Priyanka

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.