I have a small Elasticsearch cluster which is in a state where we can not start it any more.
Sequence of events:
- Start node1
- Start node2
- Set "discovery.zen.minimum_master_nodes: 2" as persistent cluster setting via REST
- Stop node2
- Adjust config file of node1 to have "discovery.zen.minimum_master_nodes: 1"
I don't have node2 available any more at all.
So now when we try to restart node1, it does not start any more because master is not elected as it still thinks it needs to see 2 nodes. Naturally it can also not see that the persistent setting is wrong and should be replaced by the newer yml-setting.
But as this is a setting of the persistent cluster state, I cannot change it unless there is a master elected, but I cannot elect a master until the setting is changed which means I am in a deadlock!
Is there a way how I can resolve this other than adding a new node just for the config-change or by deleting the cluster state file under data/cluster/nodes/0/_state?