ES nodes fail to ping with ports open and Telnet'able

Elastic Stack Elasticsearch
1

Hi there!

Weirdly, still did not find a solution to a problem.
I'm creating a new cluster right now (v6.8.6). And all nodes only see themselves.
All of them set up to be master and have discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts contains all of their IP's as well.
network.host and network.publish_host are set to their respective IP's.

In logs, I get "not enough master nodes discovered during pinging" with only 1 node listed (the one I'm listing logs on).
And in trace logs - "[o.e.d.z.UnicastZenPing ] [] [18] failed to ping" with other 2 nodes adresses.

Double checked - network connectivity is present, I can curl/telnet between them on both 9200 and 9300.

Neither SSL nor X-pack are used.
OS - Oracle Linux Server release 9.1, firewall and selinux disabled.
Tried OpenJRE 8 and Oracle JDK 17 - no difference.

Would be really grateful for some help.

2

Please share the elasticsearch.yml of all your nodes.

Also, any reason to start a new cluster with this specific old version? Elasticsearch 6.X already reached EOL and is not supported or maintained anymore.

3

Sure.

### Ingest Node ###

cluster.name: vsudi
node.name: ingest

node.data: false
node.ingest: true
node.master: true

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
thread_pool.bulk.queue_size: 1000
indices.requests.cache.size: 10%

bootstrap.memory_lock: true
network.host: 192.168.4.154
network.publish_host: 192.168.4.154

http.port: 9200
discovery.zen.minimum_master_nodes: 2

discovery.zen.ping.unicast.hosts: [ "192.168.1.154", "192.168.1.155", "192.168.1.156" ]
gateway.recover_after_nodes: 2
action.destructive_requires_name: true

logger.org.elasticsearch.discovery: TRACE



### Data Node 1 ###

cluster.name: vsudi
node.name: data1

node.data: true
node.ingest: false
node.master: true

path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
thread_pool.bulk.queue_size: 1000
indices.requests.cache.size: 10%

bootstrap.memory_lock: true
network.host: 192.168.4.155
network.publish_host: 192.168.4.155

http.port: 9200
discovery.zen.minimum_master_nodes: 2

discovery.zen.ping.unicast.hosts: [ "192.168.1.154", "192.168.1.155", "192.168.1.156" ]
gateway.recover_after_nodes: 2
action.destructive_requires_name: true

logger.org.elasticsearch.discovery: TRACE


### Data Node 2 ###

cluster.name: vsudi
node.name: data2

node.data: true
node.ingest: false
node.master: false

path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch
thread_pool.bulk.queue_size: 1000
indices.requests.cache.size: 10%

bootstrap.memory_lock: true
network.host: 192.168.4.156
network.publish_host: 192.168.4.156

http.port: 9200
discovery.zen.minimum_master_nodes: 2

discovery.zen.ping.unicast.hosts: [ "192.168.1.154", "192.168.1.155", "192.168.1.156" ]
gateway.recover_after_nodes: 2
action.destructive_requires_name: true

logger.org.elasticsearch.discovery: TRACE

I would absolutely prefer at least latest ES 7, but I'm currently moving data from another cluster and the frontend using it is not ready for the update yet.

4

Why are you using 192.168.4.* as the network.host and network.publish_host but using 192.168.1.* in the discovery settings?

Those needs to be the same network, your nodes are listening on the 192.168.4.* IP address, so your other nodes need to communicate with this IP.

2 Likes
5

Oh my god, I'm an idiot)

Automatically put the "192.168.1." part and was just missing it.

Thank you very much, I was banging my head against it for hours)
Have a great day!

2 Likes
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.