Hello,
-
I have a 3-node cluster with two nodes holding data and one as the voting node thastorees not hold any data.
-
All virtual nodes run on ESXi 8.0 with Ubuntu 22.04.1 & Elasticsearch 8.6.2.
-
OS and data are on separate SSDs.
-
One of the data nodes had it's SSD filled & VMWare ESXi was not allowing me to move ahead. Hence I replaced the SSD with an empty one hoping to have the dsuppliedlled from the remaining data node.
-
After replacing the SSD, formatting it, mounting the volume & applying for the correct owner permissions, I restarted all three nodes. I am, however, getting the following error:
[2023-02-24T11:09:54,290][WARN ][o.e.c.c.ClusterFormationFailureHelper] [primarynode] master not discovered or elected yet, an election requires at least 2 nodes with ids from [1qviKJ0KSfCe8-81Ev_Yhw, bjoclsc8SZSjjwuL0Rd_Jw, LYXT25ZWR5m7mn4uh0bjng], have only discovered non-quorum [{primarynode}{KrXCHV4OReaM1KWYk8kdVg}{jLZx0tVjRful6ARnYgs6UA}{primarynode}{192.168.0.235}{192.168.0.235:9300}{dilmr}, {secondarynode}{1qviKJ0KSfCe8-81Ev_Yhw}{3YrVbixxSg6G5IHgv9NcAA}{secondarynode}{192.168.0.236}{192.168.0.236:9300}{dilmr}]; discovery will continue using [192.168.0.236:9300, 192.168.0.237:9300] from hosts providers and [{primarynode}{KrXCHV4OReaM1KWYk8kdVg}{jLZx0tVjRful6ARnYgs6UA}{primarynode}{192.168.0.235}{192.168.0.235:9300}{dilmr}, {secondarynode}{1qviKJ0KSfCe8-81Ev_Yhw}{3YrVbixxSg6G5IHgv9NcAA}{secondarynode}{192.168.0.236}{192.168.0.236:9300}{dilmr}, {votingonlynode}{bjoclsc8SZSjjwuL0Rd_Jw}{RzpWrqWXTfiW3yKtJaS8bg}{votingonlynode}{192.168.0.237}{192.168.0.237:9300}{imrv}] from last-known cluster state; node term 1083, last-accepted version 15142102 in term 1083
[2023-02-24T11:09:56,629][INFO ][o.e.x.s.a.RealmsAuthenticator] [primarynode] Authentication of [**redacted**] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2023-02-24T11:09:56,721][INFO ][o.e.x.s.a.RealmsAuthenticator] [primarynode] Authentication of [**redacted**] was terminated by realm [reserved] - failed to authenticate user [elastic]
Here is the discovery portion of my elasticsearch.yml that I reckon requires an edit (referring to this: Master not discovered or elected yet, an election requires at least 2 nodes - #4 by li-peng)
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#discovery.seed_hosts: ["primarynode", "secondarynode"]
#discovery.seed_hosts: ["primarynode", "secondarynode", "votingonlynode"]
discovery.seed_hosts: ["192.168.0.235", "192.168.0.236", "192.168.0.237"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#cluster.initial_master_nodes: ["primarynode", "secondarynode"]
#30-05-2021: Commented as part of diagnostic since the sync was failing. That restore did not work.
#30-05-2021: Removing the comment
#30-05-2021: Needs review - added to tasks
#31-05-2021: Commenting based on: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-discovery-bootstrap-cluster.html
#-> This setting is only needed to form a cluster.
#cluster.initial_master_nodes: ["192.168.0.235", "192.168.0.236"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
How do I resolve the error?