Cluster.initial_master_nodes and discovery.seed_hosts

Elastic Stack Elasticsearch
1

Hi, I have elasticsearch cluster (8.7.0) on Kubernetes.
In the configmaps of the nodes (master,data,client) I had:

...
discovery.seed_hosts: ${NODE_LIST}
cluster.initial_master_nodes: ${MASTER_NODES}
...

In my deployments I have:

...
        - name: NODE_LIST
          value: elasticsearch-master, elasticsearch-data, elasticsearch-client
        - name: MASTER_NODES
          value: elasticsearch-master
...

After I started the cluster I deleted cluster.initial_master_nodes: ${MASTER_NODES} from every node in my cluster, but after I restarted the master node I have problems.

This is in the log of the client node:

"WARN", "message":"address [10.103.15.36:9300], node [null], requesting [false] discovery result: [][10.103.15.36:9300] connect_exception: Failed execution: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: elasticsearch-data/10.103.15.36:9300: Connection refused: elasticsearch-data/10.103.15.36:9300: Connection refused", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-client][generic][T#6]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"rliY3rQEQ1eZSvzzECFl-Q","elasticsearch.node.id":"B6L7OzUzRl60PudgYyb3eg","elasticsearch.node.name":"elasticsearch-client","elasticsearch.cluster.name":"elasticsearch"

This is in the log of data node:

"WARN", "message":"address [10.99.198.104:9300], node [null], requesting [false] discovery result: [elasticsearch-client][192.168.217.123:9300] successfully discovered master-ineligible node {elasticsearch-client}{B6L7OzUzRl60PudgYyb3eg}{GbFFdl4WRLCWCGpB-4C54A}{elasticsearch-client}{192.168.217.123}{192.168.217.123:9300}{8.7.0} at [10.99.198.104:9300]; to suppress this message, remove address [10.99.198.104:9300] from your discovery configuration or ensure that traffic to this address is routed only to master-eligible nodes", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-data][generic][T#2]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"rliY3rQEQ1eZSvzzECFl-Q","elasticsearch.node.id":"D1h6fNfURbqVHE6m9BDGuw","elasticsearch.node.name":"elasticsearch-data","elasticsearch.cluster.name":"elasticsearch"

This is in the log of master node:

"WARN", "message":"address [10.99.198.104:9300], node [null], requesting [false] discovery result: [elasticsearch-client][192.168.217.123:9300] successfully discovered master-ineligible node {elasticsearch-client}{B6L7OzUzRl60PudgYyb3eg}{GbFFdl4WRLCWCGpB-4C54A}{elasticsearch-client}{192.168.217.123}{192.168.217.123:9300}{8.7.0} at [10.99.198.104:9300]; to suppress this message, remove address [10.99.198.104:9300] from your discovery configuration or ensure that traffic to this address is routed only to master-eligible nodes", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master][generic][T#2]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"elasticsearch-master","elasticsearch.cluster.name":"elasticsearch"

I guess I am doing something wrong or something in my configuration is incorrect, what I need to change in my configuration so this cluster can work can you please help me?

2

Also in the logs of the data nodes I have this WARN, before I get the one that I posted:

"WARN", "message":"master not discovered yet: have discovered [{elasticsearch-data}{YzvQtDJ2TsyGO-uqeArzRg}{445g3DjTRYuZ6O3L9qRveQ}{elasticsearch-data}{192.168.150.95}{192.168.150.95:9300}{d}{8.7.0}, {elasticsearch-master}{_pu0iOx-TJ2N3ygjwJn7SQ}{Cj5ObIPlRaWYqTaERj0kdg}{elasticsearch-master}{192.168.217.121}{192.168.217.121:9300}{m}{8.7.0}]; discovery will continue using [10.102.87.156:9300, 10.99.198.104:9300, 10.103.15.36:9300] from hosts providers and [{elasticsearch-master}{wVhldoU0S7uUY5K690A4HA}{P4c-Q1rRTeOWg3GZNQbuqw}{elasticsearch-master}{192.168.217.99}{192.168.217.99:9300}{m}{8.7.0}] from last-known cluster state; node term 1, last-accepted version 71 in term 1; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.7/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-data][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.cluster.uuid":"ZpYLjapRS36Q7m92iMOXfA","elasticsearch.node.id":"YzvQtDJ2TsyGO-uqeArzRg","elasticsearch.node.name":"elasticsearch-data","elasticsearch.cluster.name":"elasticsearch"

And in the logs of the master node:

"WARN", "message":"master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{elasticsearch-master}{_pu0iOx-TJ2N3ygjwJn7SQ}{Cj5ObIPlRaWYqTaERj0kdg}{elasticsearch-master}{192.168.217.121}{192.168.217.121:9300}{m}{8.7.0}]; discovery will continue using [10.102.87.156:9300, 10.99.198.104:9300, 10.103.15.36:9300] from hosts providers and [{elasticsearch-master}{_pu0iOx-TJ2N3ygjwJn7SQ}{Cj5ObIPlRaWYqTaERj0kdg}{elasticsearch-master}{192.168.217.121}{192.168.217.121:9300}{m}{8.7.0}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.7/discovery-troubleshooting.html", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-master][cluster_coordination][T#1]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"elasticsearch-master","elasticsearch.cluster.name":"elasticsearch"
3

See After restarting the master node, data and client nodes cannot discover the master - #3 by daniela09