Master Not Discovered Exception : FailedToCommitClusterStateException: node is no longer master for term 55 while handling publication

Elastic Stack Elasticsearch
1

Hello Team,

Last time i checked the cluster health was on December, from that time no one monitored the servers and didn't check the cluster health.

Last week when i tried to access kibana, encountered an error "Kibana service is not yet ready"
then i checked all the services , my cluster health was RED!!!:frowning:

Cluster details :
Node A, Node B, Node C
3 Master eligible nodes + data nodes

Kibana is hosted in Node C

Below things tested, while troubleshooting the issue :

https://Node A:9200/_cat/nodes OR https://Node B:9201/_cat/nodes - returns Node A and Node B , Node C is detached from cluster

telnet Node A 9200 - from Node C - Connected
telnet Node A 9300 - from Node C - Connected
telnet Node B 9201 - from Node C - Connected
telnet Node B 9301 - from Node C - Connected

telnet Node C 9202 - from Node A - Connected
telnet Node C 9302 - from Node A - Connected
telnet Node C 9202 - from Node B - Connected
telnet Node C 9302 - from Node B - Connected

curl "https://Node A:9200" from Node B - Working
curl "https://Node B:9201" from Node A - Working

curl "https://Node A:9200" from Node C - Not Working - curl(35) TCP Connection reset by
curl "https://Node B:9201" from Node C - Not Working - curl(35) TCP Connection reset by

curl "https://Node C:9202" from Node A - Not Working - Security exception
curl "https://Node C:9202" from Node B - Not Working - Security exception.

.security index is placed in some other node, Node C not able to fetch the authentication details

I also tried to curl using insecure mode doubting problem with ssl certificates, but ended up with same problem.
Changed the ssl.verification.mode to none and restarted all 3 instances , but no luck.

There is a connectivity between these 3 servers, but Node 3 is not able to join the cluster and kibana complaining that No Living Connection at https://Node A:9200. https://Node B:9201.... really not getting the root cause!!

Getting master_not_discovered exception from Node C , and FailedToCommitClusterStateException: node is no longer master for term 55 while handling publication error from Node B log.

Can i get a private link to upload all my logs and configurations? that would help you for analysis.

Thanks in Advance,
Ramya

2

Sharing the full logs would be helpful :slight_smile:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.