I am using 6 node ES cluster of 3 masters and 3 data nodes. one master and one data node in each vm of the machine. Totally 3 machines.
replication factor = 3. num of shards = 3. node left delay being set to 5m.
When one node goes down, meaning one master and one data node goes down. ES goes to red state with shards being NODE_LEFT state. And it takes lot of time ~10min to come to yellow state. Is it expected to take this much time?
Later we reduced the num of replicas to 2 to check whether it can reduce the time.
during this time all reads and writes fail. High availability is lost. Is there a way to promote replicas to primary soon and make ES cluster available soon. delaying replica assignment can happen in background. currently it seems replica assignment is happening in sync and cluster is non responsive.