o.e.c.c.ClusterFormationFailureHelper] [Master-node] master not discovered or elected yet, an election requires a node with id [Y11KbKB-SX2HIYM77Z1Zqg], have discovered

Elastic Stack Elasticsearch
1

Hello,

I am trying to form a 3 node ES cluster but i am getting a above mentioned error.
I have one master node and 2 data nodes.

My ES configurations are as follows:

    **For master node:**
    cluster.name: InTouch
    cluster.remote.connect: false
    network.host: ServerA
    transport.host: ServerA
    transport.tcp.port: 9300-9400
    node.name: Master-node
    node.master: true
    node.data: false
    node.ingest: true
    discovery.zen.ping.unicast.hosts: [ServerA, ServerB,ServerC]
    cluster.initial_master_nodes: ["Master-node","Data-node1"]
    discovery.zen.minimum_master_nodes: 2
    bootstrap.memory_lock: true
    network.bind_host:  *local*
    path.data: /elasticsearch/data/
    path.logs: /var/log/elasticsearch/
    path.repo: /elasticsearch/repo/

**For Data node 1**:
cluster.name: InTouch
node.name: Data-node1
node.master: true
node.data: true
node.ingest: false
network.host: ServerB
discovery.zen.ping.unicast.hosts: [ServerA, ServerB,ServerC]
transport.tcp.port: 9300-9400
path.data: /elasticsearch/data/
path.logs: /var/log/elasticsearch/
path.repo: /elasticsearch/repo/

**For Data node 2**:
cluster.name: InTouch
node.name: Data-node2
node.master: false
node.data: true
network.host: ServerC
discovery.zen.ping.unicast.hosts: [ServerA, ServerB,ServerC]
transport.tcp.port: 9300-9400
path.data: /elasticsearch/data/
path.logs: /var/log/elasticsearch/
path.repo: /elasticsearch/repo/

Thanks,
Priyanka

2

You've not shared the whole log message, and the missing bit is important. Please share the full message.

3

Hello @DavidTurner,

Please find below error log:

[2020-01-06T06:26:46,981][WARN ][o.e.c.c.ClusterFormationFailureHelper] [Master-node] master not discovered or elected yet, an election requires a node with id [Y11KbKB-SX2HIYM77Z1Zqg], have discovered [{Master-node}{PcrrIC01SECgC-gDmABeSA}{sSuci3QXRSSN4JyZ1XaijQ}{ServerA}{ip_address of server A:9300}{ilm}{ml.machine_memory=51539005440, xpack.installed=true, ml.max_open_jobs=20}] which is not a quorum; discovery will continue using [ip_address of server B:9300, ip_address of server C:9300] from hosts providers and [{Master-node}{PcrrIC01SECgC-gDmABeSA}{sSuci3QXRSSN4JyZ1XaijQ}{ServerA}{ip_address of server A:9300}{ilm}{ml.machine_memory=51539005440, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 66, last-accepted version 1011 in term 66

Regards,
Priyanka

4

So Master-node is looking in vain for a node with ID Y11KbKB-SX2HIYM77Z1Zqg. It is not discovering any other nodes. I suspect that Y11KbKB-SX2HIYM77Z1Zqg is the node ID of Data-node1. Check the first few log lines on Data-node1 to find its node ID to confirm or otherwise.

I recommend fixing the port of your master-eligible nodes using transport.tcp.port: 9300. This means both Master-node and Data-node1. You are currently using 9300-9400 but if Elasticsearch picks a port other than 9300 then you need to configure discovery.zen.ping.unicast.hosts with that port instead.

Note also that in v7 you should replace discovery.zen.ping.unicast.hosts with discovery.seed_hosts, and discovery.zen.minimum_master_nodes should be removed.

Note also that there's no point in having two master-eligible nodes. I recommend setting node.master: true on all three nodes.

5

Hello @DavidTurner,

ID Y11KbKB-SX2HIYM77Z1Zqg is of data node 3. I have checked the logs for the same.

I have changed transport.tcp.port: 9300-9400 settings to only 9300 port.

I have also repplaced discovery.zen.ping.unicast.hosts with discovery.seed_hosts

I have also changed the setting for node.master: true for all the 3 nodes.

I am getting below message after run of all the 3 nodes on master node:

 [2020-01-06T12:45:41,576][INFO ][o.e.x.m.e.l.LocalExporter] [Master-node] waiting for elected master node [{Data-node2}{Y11KbKB-SX2HIYM77Z1Zqg}{UrZFy5cOQ7Ck3dBs3HEH0Q}{server C}{ip_address of server B:9300}{dlm}{ml.machine_memory=17179267072, ml.max_open_jobs=20, xpack.installed=true}] to setup local exporter[default_local] (does it have x-pack installed?)
[2020-01-06T12:45:41,826][INFO ][o.e.c.s.ClusterSettings  ] [Master-node] updating [xpack.monitoring.collection.enabled] from [false] to [true]
[2020-01-06T12:45:41,998][INFO ][o.e.l.LicenseService     ] [Master-node] license [abb78209-b323-4d4a-8782-a8f7eaf744d4] mode [basic] - valid
[2020-01-06T12:45:42,014][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [Master-node] Active license is now [BASIC]; Security is disabled
[2020-01-06T12:45:42,295][INFO ][o.e.x.s.a.TokenService   ] [Master-node] refresh keys
[2020-01-06T12:45:42,529][INFO ][o.e.x.s.a.TokenService   ] [Master-node] refreshed keys

Is this correct? means can i say now ES cluster is up and running?

Regards,
Priyanka

6

Yes that looks better.

7

Hello @DavidTurner,

Thanks for update.
I have one more question.
All 3 nodes are master eligible. So if one goes down other should be eligible to elect itself as a master node and will form a 2 node cluster.
How i can check this with my cluster?
Can you please guide me on this more?

Regards,
Priyanka