Unable to communicate between 2 master nodes creating cluster issue

[devarajsit]

I tried creating a new Elasticsearch cluster with 3 master nodes, 3 data nodes and 2 clients. All the 3 master nodes status is in Running but when i verified the logs of the third node, it says time out connecting to 1st node:

{"type": "server", "timestamp": "2023-03-30T11:34:24,460Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "master not discovered or elected yet, an election requires at least 2 nodes with ids from [gp82BsoETGyQxrVuq4KUDg, va3LZXpBTe6m2pdcE0nqvg, CsWX-u7XR8u2jNQ4AeD1Eg], have discovered possible quorum [{elasticsearch-master-2}{va3LZXpBTe6m2pdcE0nqvg}{LuzbJdgQTOOVnXQO7NRW_w}{10.237.14.89}{10.237.14.89:9300}{ilmr}, {elasticsearch-master-1}{gp82BsoETGyQxrVuq4KUDg}{o0I5HcIfTBmFDXUJiWqcng}{10.236.204.54}{10.236.204.54:9300}{ilmr}]; discovery will continue using [10.236.204.54:9300, 10.233.173.202:9300] from hosts providers and [{elasticsearch-master-2}{va3LZXpBTe6m2pdcE0nqvg}{LuzbJdgQTOOVnXQO7NRW_w}{10.237.14.89}{10.237.14.89:9300}{ilmr}] from last-known cluster state; node term 5, last-accepted version 59 in term 5" }
{"type": "server", "timestamp": "2023-03-30T11:34:26,391Z", "level": "WARN", "component": "o.e.d.PeerFinder", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "address [10.233.173.202:9300], node [null], requesting [false] connection failed: [][10.233.173.202:9300] connect_timeout[30s]" }

When i executed the curl to cat nodes. i can see 2 nodes are there as part of the cluster:

10.225.3.47 3 75 0 0.00 0.02 0.03 cdfhilrstw - elasticsearch-data-1
10.233.173.204 10 75 0 0.18 0.09 0.02 cdfhilrstw - elasticsearch-data-0
10.233.173.202 38 83 1 0.18 0.09 0.02 ilmr * elasticsearch-master-0
10.236.204.54 64 82 1 0.00 0.02 0.02 ilmr - elasticsearch-master-1

elasticsearch-master-2 and elasticsearch-data-2 are not joining to cluster. Any help would be appreciated.

[Christian_Dahlqvist]

It would help if you shared which version of Elasticsearch you are running and the configuration of the nodes.

[devarajsit]

Hi Christian,

Thanks for replying.

Version is "7.17.1"
Let me know what configuration you need for the nodes?

  cluster.name: elasticsearch
    action.auto_create_index: true
    xpack.security.enabled: false
    xpack.security.transport.ssl.enabled: false
    node.data: ${NODE_DATA:true}
    node.master: ${NODE_MASTER:true}
    # vm.max_map_count: 262144
    node.name: ${HOSTNAME}

    network.host: 0.0.0.0
    http.host: 0.0.0.0

    # see https://github.com/elastic/elasticsearch-definitive-guide/pull/679
    processors: ${PROCESSORS:}

    # avoid split-brain w/ a minimum consensus of two masters plus a data node
    gateway.expected_master_nodes: ${EXPECTED_MASTER_NODES:1}
    gateway.expected_data_nodes: ${EXPECTED_DATA_NODES:1}
    gateway.recover_after_time: ${RECOVER_AFTER_TIME:5m}
    gateway.recover_after_master_nodes: ${RECOVER_AFTER_MASTER_NODES:1}
    gateway.recover_after_data_nodes: ${RECOVER_AFTER_DATA_NODES:1}

[Christian_Dahlqvist]

How are you bootstrapping the cluster?

Are you not setting discovery.seed_hosts or cluster.initial_master_nodes for any of the nodes?

I would also recommend running 3 master eligible nodes as having just 2 does not give you any high availability.

[devarajsit]

In the configuration we have discovery.seed_hosts which is a kubernetes service and will take all the three nodes

   - name: discovery.seed_hosts
          value: elasticsearch-discovery
========================
elasticsearch-discovery                 10.233.173.207:9300,10.236.204.55:9300,10.237.14.92:9300                3h22m

I have modified the master from 2 to 1 since i was getting issue with cluster formation.