3 node cluster failure

Elastic Stack Elasticsearch
1

Hi,

I am new to elasticsearch. I have created a 3 node elasticsearch cluster. All master and data set to true. All 3 nodes are running:
Centos 8,
ES version 7.8.1
Java - openjdk 11.0.8 2020-07-14 LTS.
Firewall - in public, with the IP address' of the other 2 nodes in the source, ports 9200-9400/tcp open
(I changing the default zone to Trusted, and also disabling the firewall all together)

elasticsearch.yml:
cluster.name: es-cf
node.name: es-01
node.master: true
node.data: true
path.data: /data/esdata
path.logs: /data/eslog
network.host: 192.168.105.121
network.publish_host: 192.168.105.121
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["192.168.105.121:9200", "192.168.105.122:9200","192.168.105.123:9200"]
discovery.zen.minimum_master_nodes : 2
cluster.initial_master_nodes: ["es-01", "es-02", "es-03"]
gateway.recover_after_master_nodes: 2

service starts fine, but getting the following error in the logs:
[o.e.c.c.ClusterFormationFailureHelper] [es-01] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [es-01, es-02, es-03] to bootstrap a cluster: have discovered [{es-01}{HQLi2hM9TtOkMDUiRScjIA}{eUuOsUU5Rx-HT9mRp3sV2g}{192.168.105.121}{192.168.105.121:9300}{dilmrt}{ml.machine_memory=16590868480, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [192.168.105.121:9200, 192.168.105.122:9200, 192.168.105.123:9200] from hosts providers and [{es-01}{HQLi2hM9TtOkMDUiRScjIA}{eUuOsUU5Rx-HT9mRp3sV2g}{192.168.105.121}{192.168.105.121:9300}{dilmrt}{ml.machine_memory=16590868480, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

I think the error must be in my elasticsearch.yml file, as i turned the firewall off completly on the 3 nodes and it made no difference. Any advice would be greatly appreciated

thanks

Ian

2

I think i have fixed this, from a suggestion post below.

changed the discovery.seed_hosts to port 9300 rather than 9200 that I had. All c

3

Welcome to the forum. Questions:

  • Shouldn't the seed hosts be port 9300? You have 9200, but shouldn't put a port here unless it's unusual; port 9200 is for REST API operations, but the cluster internally works on the transport interface, usually port 9300 - remove your :9200 and I bet it'll be happy.

Minor things:

  • Are you sure you are using the JVM 11, not the JVM bundled with V7?
  • What does 'in public' mean on a firewall, as public & elasticsearch don't go well together, security wise.
  • Why set network.publish_host as it'll use network.host?
4

As per documentation you should remove port number if you use the default 9300

Citation Out of the box, without any network configuration, Elasticsearch will bind to the available loopback addresses and will scan local ports 9300 to 9305 to try to connect to other nodes running on the same server. This provides an auto-clustering experience without having to do any configuration.

Dominique

1 Like
5

changing port 9300 has indeed resolved the issue.

I will remove it completely if that is the recommended setup.
I set network.publish_host just in an effort to force things to works as I would have expected when looking through the ES documentation, again this is something I can remove, however currently it is all working so might not bother.

I have managed to send some winlogbeats into my elasticsearch cluster, what is the issue using the firewall-cmd public zone?

thanks

6

Glad it worked; I just didn't understand your public FW comments - you generally do not want an ES server on the public Internet or else anyone can query it, steal your data, and so on. Usually it's very protected, and security x-pack turned on - this can be a problem if you are sending data over the Internet, and is you'll often see Logstash or something public, but don't let that port 9200 be public without real protection.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.