Ha in two node elasticsearch

Elastic Stack Elasticsearch
1

Hi,

Currently, we setup two node elasticsearch cluster in my lab and configured ha. We set the both the nodes are master and data. but when the elk01 master node is down, the elk02 doesnt take the leader process.

ELK02 node LOG :

[2023-04-26T05:53:24,772][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elk02] master not discovered or elected yet, an election requires a node with id [bwJj1LCyTN2Abv_OdGuI1g], have only discovered non-quorum [{elk02}{Auut4OPBRWCJVZF8HTVDnA}{-g_C18DMTW-krUVX8HCD1Q}{elk02}{192.168.170.36}{192.168.170.36:9300}{dm}{8.7.0}]; discovery will continue using [192.168.170.7:9300] from hosts providers and [{elk02}{Auut4OPBRWCJVZF8HTVDnA}{-g_C18DMTW-krUVX8HCD1Q}{elk02}{192.168.170.36}{192.168.170.36:9300}{dm}{8.7.0}] from last-known cluster state; node term 6, last-accepted version 93 in term 6; for troubleshooting guidance, see Troubleshooting discovery | Elasticsearch Guide [8.7] | Elastic
[2023-04-26T05:53:24,852][WARN ][o.e.d.PeerFinder ] [elk02] address [192.168.170.7:9300], node [null], requesting [false] discovery result: [192.168.170.7:9300] connect_exception: Failed execution: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: elk01/192.168.170.7:9300: Connection refused: elk01/192.168.170.7:9300: Connection refused
[2023-04-26T05:53:25,854][WARN ][o.e.d.PeerFinder ] [elk02] address [192.168.170.7:9300], node [null], requesting [false] discovery result: [192.168.170.7:9300] connect_exception: Failed execution: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: elk01/192.168.170.7:9300: Connection refused: elk01/192.168.170.7:9300: Connection refused
[2023-04-26T05:53:26,853][WARN ][o.e.d.PeerFinder ] [elk02] address [192.168.170.7:9300], node [null], requesting [false] discovery result: [192.168.170.7:9300] connect_exception: Failed execution: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: elk01/192.168.170.7:9300: Connection refused: elk01/192.168.170.7:9300: Connection refused
[2023-04-26T05:53:27,854][WARN ][o.e.d.PeerFinder ] [elk02] address [192.168.170.7:9300], node [null], requesting [false] discovery result: [192.168.170.7:9300] connect_exception: Failed execution: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: elk01/192.168.170.7:9300: Connection refused: elk01/192.168.170.7:9300: Connection refused

2

This is expected. You need at least 3 nodes to achieve HA as Elasticsearch requires a strict majority of nodes to be available in order to elect a master and be fully functioning. See this section in the docs for more information.

1 Like
3

Hi Christian,

If its possible can we manually change the elasticsearch master , when the node is gone ?

4

No, there is no safe way to do that. There is a tool that can help you reconfigure the cluster but it is not safe and can result in data loss. If you want HA you should not rely on this and instead add a third node. Note that the third node can be a smaller dedicated master node.

5

Hi Christian,

Then i plan for four nodes elasticsearch. two nodes will be a master and data.

Another two node will be only master. Now, if the two nodes, 1 master only node and 1 data and master node is down can i use the elasticsearch right ?

6

If you have 4 master eligible nodes you need 3 (majority) to be available in order to elect a master. I would therfore go for 3 nodes on 3 different hosts.

7

Hi Christian ,

But here we have only 2 hosts. Let me know how to configure ?

8

You can not have a fully HA cluster with just 2 hosts. You need a small third host. There is no way around this.

9

Hi Christian,

If we create 5 nodes elasticsearch, if we down 3 nodes also its working fine right ?

10

No. If you have 5 master eligible nodes you need 3 to be available in order to form a majority. In this case you can only afford to lose 2 nodes.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.