Unable to connect to secondary master node, when manually stopping primary master node

Elastic Stack Elasticsearch
1

I got a small project so thought to go with only 2nodes ES cluster and maintain HA. Both nodes are act as Master and data node, i wanted to maintain any 1 node should be up all the time. Both machine having the same config and working fine when both nodes are available. But when i down node1, node2 showing master node not discovered. Can someone help to fix this?

###################################
my elasticsearch.yml file

cluster.name: abc-escluster
node.name: abc-es2
node.master: true
node.data: true
node.ingest: true
network.host: 0.0.0.0
discovery.seed_hosts: ["172.18.18.160","172.18.18.170"]
discovery.zen.minimum_master_nodes: 2
cluster.initial_master_nodes: ["172.18.18.160","172.18.18.170"]
node.voting_only: false
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.monitoring.collection.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "http.p12"

log from node2 looks like:

########################
[2022-03-29T06:58:49,182][WARN ][o.e.c.c.ClusterFormationFailureHelper] [abc-es2] master not discovered or elected yet, an election requires a node with id [gv95oj0pQbK-AAKzHtyo2w], have discovered [{vcci-es2}{owfHWxOiQ1OrJv7urv-nsA}{F6GLd6mqT7a5fyoAguXuOA}{172.18.18.170}{172.18.18.170:9300}{dilmrt}{ml.machine_memory=4067299328, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] which is not a quorum; discovery will continue using [172.18.18.160:9300] from hosts providers and [{vcci-es2}{owfHWxOiQ1OrJv7urv-nsA}{F6GLd6mqT7a5fyoAguXuOA}{172.18.18.170}{172.18.18.170:9300}{dilmrt}{ml.machine_memory=4067299328, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 11, last-accepted version 342 in term 11
[2022-03-29T06:58:49,277][INFO ][o.e.x.s.a.AuthenticationService] [vcci-es2] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
[2022-03-29T06:58:49,381][INFO ][o.e.x.s.a.AuthenticationService] [vcci-es2] Authentication of [elastic] was terminated by realm [reserved] - failed to authenticate user [elastic]
##########################

I suspect, have created my elastic user in node1, As i have stopped the node1, the user credentials are not getting validated in node2.

How can i maintain the same state with 2node cluster. If any1 node goes down still my es dashboard shouldnt come down.

Regards,
Dhamodharan.

2

You can not have a highly available cluster with only 2 nodes. Elasticsearch requires a majority of master eligible nodes to be available for a master to be elected, which means both nodes must be available in a two node cluster. You will need a minimum of 3 nodes for HA and could add a small voting-only dedicated master node to act as a tiebreaker. Please see this section in the docs for details.

3

Thanks for the information, thats fine. kindly suggest how can i bring the sec node as master when primary node down. I dont want to go with more nodes. I manually stopped node1 and tried to pullup node2, but node2 unable to validate the elastic user, as its created in node1. How can i manually bring node2 as master.
Considering node2 as data node initially, during failure i wanna make node2 as master.

4

You can not. Three nodes are required.

5

I have configured 3nodes to make it HA and its perfectly working fine. Now my challenge is how i can i get the endpoint of the cluster for the application configuration, do we have any config to enable the dns/endpoint for the cluster?? Sharing individual host name is not a solution. Else can i go with NLB from aws??

6

Most clients accept a list of hostnames and support load balancing and failover across these. If you require a single endpoint you probably need to put a load balancer in front of the cluster.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.