Hi Community,
I have cluster of 8 nodes ( 3 master + 5 data) and configured replica as 4 for each and every index in order to keep all shards in each and every node(can sustain max of 4 data nodes failure). Now in process of full cluster restart, I've shutdown all data + master nodes. I started restarting cluster by making master nodes up first then data nodes 1 by one. As I had replica 4, my cluster should have recovered all indices from data node-1 when I restarted it. But it didn't happened, cluster state was red, and all shards were in unassigned state(I didn't find any thing in logs like why shards recovering is not happening). Instead when I changed replication to <=1, all shards are started recovering and finally my cluster state went to yellow .
Can you please let me know if this is expected behaviour/I'm missing any configuration here or bug in elasticsearch.