Last time i checked the cluster health was on December, from that time no one monitored the servers and didn't check the cluster health.
Last week when i tried to access kibana, encountered an error "Kibana service is not yet ready"
then i checked all the services , my cluster health was RED!!!
Cluster details :
Node A, Node B, Node C
3 Master eligible nodes + data nodes
Kibana is hosted in Node C
Below things tested, while troubleshooting the issue :
telnet Node A 9200 - from Node C - Connected
telnet Node A 9300 - from Node C - Connected
telnet Node B 9201 - from Node C - Connected
telnet Node B 9301 - from Node C - Connected
telnet Node C 9202 - from Node A - Connected
telnet Node C 9302 - from Node A - Connected
telnet Node C 9202 - from Node B - Connected
telnet Node C 9302 - from Node B - Connected
.security index is placed in some other node, Node C not able to fetch the authentication details
I also tried to curl using insecure mode doubting problem with ssl certificates, but ended up with same problem.
Changed the ssl.verification.mode to none and restarted all 3 instances , but no luck.
There is a connectivity between these 3 servers, but Node 3 is not able to join the cluster and kibana complaining that No Living Connection at https://Node A:9200. https://Node B:9201.... really not getting the root cause!!
Getting master_not_discovered exception from Node C , and FailedToCommitClusterStateException: node is no longer master for term 55 while handling publication error from Node B log.
Can i get a private link to upload all my logs and configurations? that would help you for analysis.
Thanks in Advance,