Setting discovery.zen.minimum_master_nodes via REST can cause unstartable cluster

Elastic Stack Elasticsearch
1

Hi,

I have a small Elasticsearch cluster which is in a state where we can not start it any more.

Sequence of events:

  • Start node1
  • Start node2
  • Set "discovery.zen.minimum_master_nodes: 2" as persistent cluster setting via REST
  • Stop node2
  • Adjust config file of node1 to have "discovery.zen.minimum_master_nodes: 1"

I don't have node2 available any more at all.

So now when we try to restart node1, it does not start any more because master is not elected as it still thinks it needs to see 2 nodes. Naturally it can also not see that the persistent setting is wrong and should be replaced by the newer yml-setting.

But as this is a setting of the persistent cluster state, I cannot change it unless there is a master elected, but I cannot elect a master until the setting is changed which means I am in a deadlock!

Is there a way how I can resolve this other than adding a new node just for the config-change or by deleting the cluster state file under data/cluster/nodes/0/_state?

Thanks... Dominik.

2

The API settings take precedence over the config file ones. And persistent API ones over transient.
You need to add another node to get this back to a working state.

However you are right in that this causes issues. Would you mind opening an issue on github so we can discuss how we can make this process better?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.