I have four ES systems( Let's assume that each of the four node names is A, B, C, and D).
A, B, C, and D is master and data node.
Last week I did rolling upgrade all of ES System from 6.3 to 6.8. It was perfect!
This week I tried rolling upgrade all from 6.8 to 7.1.1.
A , B and C was good. There was no problem. It was clear.
But there's a problem when I tried to upgrade D. D was an elected master.
I set a config file like this. I guess this problem is not with this.
discovery.seed_hosts: ["172.31.xx.A", "172.31.xx.B", "172.31.xx.C", "172.31.xx.D"]
cluster.initial_master_nodes: ["node-C", "node-D"]
I stopped the elasticsearch process on D for rolling upgrade. I was watching Kibana monitoring continuously. D was still on the screen with the elected master sign. I was waiting for some minutes, D was still alive. I had no choice but restart D of 7.1.1 version.
This error was displayed on the screen then.
[WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-D] master not discovered or elected yet, an election requires at least 2 nodes with ids from ...
D was still on the screen with the elected master sign yet.
You know, there is no way to change cluster master forcibly. So, I restart A, B, and C one by one. (I know THAT was too stupid.)
But all system was displayed the same error on the screen.
What's the way to get this back on track?