Unable to communicate between 2 master nodes creating cluster issue

I tried creating a new Elasticsearch cluster with 3 master nodes, 3 data nodes and 2 clients. All the 3 master nodes status is in Running but when i verified the logs of the third node, it says time out connecting to 1st node:

{"type": "server", "timestamp": "2023-03-30T11:34:24,460Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "master not discovered or elected yet, an election requires at least 2 nodes with ids from [gp82BsoETGyQxrVuq4KUDg, va3LZXpBTe6m2pdcE0nqvg, CsWX-u7XR8u2jNQ4AeD1Eg], have discovered possible quorum [{elasticsearch-master-2}{va3LZXpBTe6m2pdcE0nqvg}{LuzbJdgQTOOVnXQO7NRW_w}{}{}{ilmr}, {elasticsearch-master-1}{gp82BsoETGyQxrVuq4KUDg}{o0I5HcIfTBmFDXUJiWqcng}{}{}{ilmr}]; discovery will continue using [,] from hosts providers and [{elasticsearch-master-2}{va3LZXpBTe6m2pdcE0nqvg}{LuzbJdgQTOOVnXQO7NRW_w}{}{}{ilmr}] from last-known cluster state; node term 5, last-accepted version 59 in term 5" }
{"type": "server", "timestamp": "2023-03-30T11:34:26,391Z", "level": "WARN", "component": "o.e.d.PeerFinder", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "address [], node [null], requesting [false] connection failed: [][] connect_timeout[30s]" }

When i executed the curl to cat nodes. i can see 2 nodes are there as part of the cluster: 3 75 0 0.00 0.02 0.03 cdfhilrstw - elasticsearch-data-1 10 75 0 0.18 0.09 0.02 cdfhilrstw - elasticsearch-data-0 38 83 1 0.18 0.09 0.02 ilmr * elasticsearch-master-0 64 82 1 0.00 0.02 0.02 ilmr - elasticsearch-master-1

elasticsearch-master-2 and elasticsearch-data-2 are not joining to cluster. Any help would be appreciated.

It would help if you shared which version of Elasticsearch you are running and the configuration of the nodes.

Hi Christian,

Thanks for replying.

Version is "7.17.1"
Let me know what configuration you need for the nodes?

  cluster.name: elasticsearch
    action.auto_create_index: true
    xpack.security.enabled: false
    xpack.security.transport.ssl.enabled: false
    node.data: ${NODE_DATA:true}
    node.master: ${NODE_MASTER:true}
    # vm.max_map_count: 262144
    node.name: ${HOSTNAME}


    # see https://github.com/elastic/elasticsearch-definitive-guide/pull/679
    processors: ${PROCESSORS:}

    # avoid split-brain w/ a minimum consensus of two masters plus a data node
    gateway.expected_master_nodes: ${EXPECTED_MASTER_NODES:1}
    gateway.expected_data_nodes: ${EXPECTED_DATA_NODES:1}
    gateway.recover_after_time: ${RECOVER_AFTER_TIME:5m}
    gateway.recover_after_master_nodes: ${RECOVER_AFTER_MASTER_NODES:1}
    gateway.recover_after_data_nodes: ${RECOVER_AFTER_DATA_NODES:1}

How are you bootstrapping the cluster?

Are you not setting discovery.seed_hosts or cluster.initial_master_nodes for any of the nodes?

I would also recommend running 3 master eligible nodes as having just 2 does not give you any high availability.

In the configuration we have discovery.seed_hosts which is a kubernetes service and will take all the three nodes

   - name: discovery.seed_hosts
          value: elasticsearch-discovery
elasticsearch-discovery       ,,                3h22m

I have modified the master from 2 to 1 since i was getting issue with cluster formation.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.