Need advice to understand cluster behavior

Elastic Stack Elasticsearch
1

Dear all.

I've got a POC with Elasticsearch 6.4.0
5 Data Nodes
2 Coordinating nodes
1 Master Only Node

In these 5 data nodes, i've got two of them master elligible.
In elasticsearch.yml, discovery.zen.minimu_master_nodes is set to 2
The master node is master only (no data, no other role)
Flat network in one site

When the master only node is really the master, i tried to shutdown the two other master eligible data nodes.
In this case, the cluster becomes red, Kibana (connected to Coordinatiing nodes) does not update any status.
I can read in the elasticsearch log from the master that it complains "Not enough master nodes discovered during pinging"...

My questions are :

  • why the cluster cannot survive in this situation since i've got a master ?
  • a master only node is really a good idea ?

Thx in advance

2

In order to elect a master node, a majority of master-eligible nodes need to be available. When you shut down two of the three master-eligible nodes, the existing master no longer has mandate to stay as master, which is why the cluster goes red.

1 Like
3

Thanks Christian.
This makes sense.

Another point is that i have 2 shards (P + R) for metricbeat. Shuting down two nodes may drive me in lost of indexes.

My conslusion follwing these tests are (in the case of two nodes shutdown at the same time):

  • select all other data nodes to master elligible (to withstand lost of the actual two master elligible nodes)
  • increase nb of shards in metricbeat to three , for the same reason.

Do you think my conclusion makes sense ?

thx in advance

Francois

4

You can only handle immediate loss of as many nodes as you have replicas as one copy must be left in the cluster to recover from.

Yes, you can make all 5 nodes master-eligible and set minimum_master_nodes to 3 instead, which will allow you to handle 2 master-eligible nodes going down.

Increasing the number of replicas to 2 (3 shard copies) will also help you handle this better.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.