Hi, I have a question related to replication. I have a cluster of 7 elastic search machines with number of replicas defined as 2. machines are amazon aws instances. I had to shutdown one instance and the reallocation of about 2000 shards to other instances took almost 10 hours. So, the question I have - is there any documentation, explaining how this works? As far as I understand the replica shards are already sitting on the nodes, which are stayed alive. What the process of "reallocation" actually does?
Thank you in advance
Unless you lower the replica count for indexes, Elasticsearch will still try to uphold the originally requested replica count.
In other words, when you shut down the node and its 2000 shards Elasticsearch did indeed promote one of the existing replicas of each index to a primary shard but then proceeded to create a new replica of the same shard since the index was still configured to have two replicas.
If you temporarily want to disable shard allocations you can do that with the cluster.routing.allocation.enable setting. This is quite useful if you're temporarily taking down a node and want to avoid hours of useless shard rebalancing.