Prioritized a master node in ES cluster

Elastic Stack Elasticsearch
1

Hello,

My ES Cluster contains 3 Master nodes (node-1, node-2, node-3), and nodes have a priority (99,98,97) in order so when node-1 goes down it elects node-2 as the new master node this is good till now, but when node-1 go up again and try to down node-2, it elects node-3 as the new master, not node-1 although the node-1 has the higher priority so what should I do to make it elect node-1 as the new master again?

my ES cluster nodes configurations:

node-1:

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: ELK
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1

node.roles: [ master, data ]

ingest.geoip.downloader.enabled: false
#
# Add custom attributes to the node:
node.attr.priority: 99
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /server/elasticsearch/data
#
# Path to log files:
#
path.logs: /server/elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.168.x.y", "192.168.x.y","192.168.x.v"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1","node-2","node-3"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*, .*
xpack.security.enabled: false

node-2:

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: ELK
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-2

node.roles: [ master, data ]

ingest.geoip.downloader.enabled: false
#
# Add custom attributes to the node:
node.attr.priority: 98
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /server/elasticsearch/data
#
# Path to log files:
#
path.logs: /server/elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.168.x.y", "192.168.x.y","192.168.x.v"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1","node-2","node-3"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*, .*
xpack.security.enabled: false

node-3:

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: ELK
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-3

node.roles: [ master, data ]

ingest.geoip.downloader.enabled: false
#
# Add custom attributes to the node:
node.attr.priority: 97
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /server/elasticsearch/data
#
# Path to log files:
#
path.logs: /server/elasticsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["192.168.x.y", "192.168.x.y","192.168.x.v"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1","node-2","node-3"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*, .*
xpack.security.enabled: false

thanks in advance.

2

Why would this attribute affect which node is elected master? Can you please point to where in the docs this is described?

3

thanks for your reply,

I found it somewhere but I can't find the URL right now, and also when asked Chat-gpt for help he suggested it to me too. and when started the cluster didn't get any errors so I thought it is not a wrong configuration line, if it is not affecting it could please help me with what should I do to make the scenario above work efficiently?

thanks in advance

4

I have never heard of this, which is why I was asking. As far as I know there is no way to force this behaviour. You can assign any attributes to nodes without causing errors. If not in use they will just be ignored.

I would not trust this. You can find numerous examples here on the forum where incorrect or ambigous advise has been given.

5

oh okay thanks for your help, I have another question idk I should open a new topic or not, if I have 2 clusters each cluster has 3 nodes, could I make the first one "the main" and the other the "DR cluster" so other applications like logstash and kibana don't connect to DR if any node of the main cluster still up and just connect if all nodes of the main cluster down?
thanks again for your help really i appreciate it

6

What are you trying to achieve? What are the requirements?

7

I am trying to monitor the wso2 API manager APIs, I could successfully monitor the API with one cluster the main, but what I am trying doing it is if the main cluster is down I want to be able still to monitor the API with the DR cluster, and when the main cluster comes up again kibana and logstash connect to the main cluster again

8

That is not exactly how it works so you may be best of setting up a single cluster that is resilient. Set up 3 nodes where all nodes are master eligible and can hold data. If you are setting up in the cloud, make sure to spread the nodes out across availability zones.

If you have 2 clusters like you describe you typically feed them all data in parallel, e.g. using a message queue. That way both clusters have the full data set and the remaining cluster can continue receiving data and serving queries if the main one goes down. The message queue will buffer data, which allows the cluster that was temporarily unavailable to catch up once back online. Unless you use a load balancer Kibana typically connects to just one node so will automatically not fail over across clusters. Logstash does not do this either.

1 Like
9

To confirm: the node.attr.priority attribute has no meaning to Elasticsearch. ChatGPT is (at best) a useful way to generate plausible-sounding fiction.

2 Likes
10

thank you for trying to help me, but is there any way to do this scenario?

11

No, as far as i know there is not. Why do you feel you need to control this? If you have 3 nodes with the same specification and configuration it does not matter which one is the master at any point in time.

12

because now I have two solutions :
1- to add the main nodes and DR nodes in one cluster so in this situation the requirement is not to elect any of the DR nodes as the master while at least one of the main nodes is up

2- The second I explained above and you thankfully suggested a solution to me

so logically to have all nodes in one cluster is more simple and reliable than 2 clusters, that is why I am trying to find something help me do it

13

Stretching a cluster across 2 data centers is probably not a solution. All nodes will be in the same cluster and you will need an absolute majority of master eligible nodes available, e.g. 4 out of 6. You can not lose half of these nodes and still have a fully funcional cluster. Nodes within a cluster also relies on low letency and efficient networking, which can sometimes be difficult to achieve when stretchinga cluster across multiple data centers.

14

Yes I think this is all covered by these docs:

Specifically:

If you’ve divided your cluster into zones, the network connections within each zone are typically of higher quality than the connections between the zones. Ensure the network connections between zones are of sufficiently high quality. You will see the best results by locating all your zones within a single data center with each zone having its own independent power supply and other supporting infrastructure. You can also stretch your cluster across nearby data centers as long as the network interconnection between each pair of data centers is good enough.

and:

If you have two zones, you should have a different number of master-eligible nodes in each zone so that the zone with more nodes will contain a majority of them and will be able to survive the loss of the other zone. For instance, if you have three master-eligible nodes then you may put all of them in one zone or you may put two in one zone and the third in the other zone. You should not place an equal number of master-eligible nodes in each zone.

1 Like
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.