Elastic Search 7.17.9 ClusterFormationFailure

Elastic Stack Elasticsearch
1

I am facing some issues in my elasticsearch cluster related to Cluster Formation with 2 nodes.
I'm trying to upgrade from elasticsearch from 7.17.0 to 7.17.9. Node1 is set as master and Node2 is not.

I stopped elasticsearch from 7.17.0 and started 7.17.9 in tha same server.

This are the errors:
ClusterFormationFailure
non-master-eligible node found

Logs of liferaydev1 about Master node

[2023-07-07T19:24:44,373][TRACE][o.e.d.HandshakingTransportAddressConnector] [liferaydev1] [connectToRemoteMasterNode[192.168.253.19:9300]] opening probe connection
[2023-07-07T19:24:44,377][DEBUG][o.e.d.PeerFinder ] [liferaydev1] address [192.168.253.19:9300], node [null], requesting [false] connection failed
org.elasticsearch.transport.ConnectTransportException: [192.168.253.19:9300] connect_exception

[2023-07-08T19:00:59,129][WARN ][o.e.d.PeerFinder] [liferaydev1] address [192.168.253.19:9300], node [null], requesting [false] connection failed: [liferaydev2][192.168.253.19:9300] non-master-eligible node found
[2023-07-08T19:00:59,428][WARN ][o.e.c.c.ClusterFormationFailureHelper] [liferaydev1] master not discovered or elected yet, an election requires one or more nodes that have already participated as master-eligible nodes
in the cluster but this node was not master-eligible the last time it joined the cluster, have only discovered non-quorum [{liferaydev1}{3DpR5D2oSVaiS3_ViNr4lw}{4NWhFQBCRRCjrQ3I-7WPQw}{192.168.253.18}{192.168.253.18:9300}{dmr}]; discovery
will continue using [192.168.253.19:9300] from hosts providers and [{liferaydev1}{3DpR5D2oSVaiS3_ViNr4lw}{4NWhFQBCRRCjrQ3I-7WPQw}{192.168.253.18}{192.168.253.18:9300}{dmr}] from
last-known cluster state; node term 5, last-accepted version 770 in term 5

[2023-07-08T09:07:54,564][WARN ][o.e.c.c.ClusterFormationFailureHelper] [liferaydev2] master not discovered yet: have discovered
[{liferaydev2}{3DpR5D2oSVaiS3_ViNr4lw}{iAl8qK0gTdiYAX9-hiLS1Q}{192.168.253.19}{192.168.253.19:9300}{d}, {liferaydev1}{3DpR5D2oSVaiS3_ViNr4lw}{JgXAgEsBSXSKY5KgpAGxsg}{192.168.253.18}{192.168.253.18:9300}{dm}]; discovery will continue using [192.168.253.18:9300] from hosts providers and
from last-known cluster state; node term 5, last-accepted version 770 in term 5

[2023-07-08T18:41:29,394][WARN ][o.e.d.PeerFindee] [liferaydev1] address [192.168.253.19:9300], node [null], requesting [false] connection failed: [liferaydev2][192.168.253.19:9300] non-master-eligible node found

As info....in node1 server, I have the cluster name as same in version 7.17.9 and the old 7.17.0

Any help is greatly appreciated.

Thank you.
SSRR

2

These logs say that liferaydev1 was not master-eligible the last time it joined a cluster, and liferaydev2 is not currently master-eligible. You need a master-eligible node that was previously master-eligible in order to form a cluster.

3

The node to was failing to join the cluster and not recognized as master node.
So, I ran the commands for Detaching nodes from their cluster.
Unsafe-bootstarp of dev1 and "detach-cluster" of dev2 from cluster.

Then added the master node first to the cluster. Once master node is successful, I started the second node which is a non-master node. I did not update anything significantly in dev1 and dev2 other than detaching nodes from cluster.

Thank you so much for your help.
SSRR

4

The detach-cluster command is not safe (risks data loss) and should not be needed unless something else has gone badly wrong in your cluster.

5

David,
I tried several options to have the dev1 server recognized as master in the cluster. For example, I changed the cluster name of Elasticsearch with a new name and master node did not work.
Since this is a new dev environment, I thought it is okay to detach the cluster.

Thank you.
SSRR

6

That's fine, but I don't want the next reader to follow you along this unsafe path without sufficient warning. Something's wrong with your setup if it needs the detach-cluster command.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.