How to avoid initializing_shards during restarts

Elastic Stack Elasticsearch
1

Hi Folks, so we have 3 node ElastichSearch cluster, when we do rollout restart we see a case like there are shards which was primary on restarted node A and now the replica on other node B promoted as primary and we see new replica started to be initializing on node C and this initializing can take very long time, is there any way to avoid new replica initializiation on node C but reuse the old primary on node A as starting point for new replica?

2

Hey.

Have a look at Upgrade Elasticsearch | Elastic Docs (step 1):

  1. Disable shard allocation.When you shut down a data node, the allocation process waits for index.unassigned.node_left.delayed_timeout (by default, one minute) before starting to replicate the shards on that node to other nodes in the cluster, which can involve a lot of I/O. Since the node is shortly going to be restarted, this I/O is unnecessary. You can avoid racing the clock by disabling allocation of replicas before shutting down data nodes:
PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}
3

@dadoonet yes checked it and doing that but right after I revert this change when node healthy I see replica shards which existed on this restarted node are re-assigned back to it and all good but initializiation still happens for new replicas on 3rd node but I would prefer elasticsearch to use old primary shard from node A as new replica foundation which should be much faster than allocating new replica to 3rd node.

Note: I see if we do dirty restart without waiting for green status cluster state restores to green fast, looks like no replica initializations happen to different node (all replicas stay on their node), but if we wait green status between restarts then because primary changes to different node and new replica initialises things takes very, very long time and my question is even that new replica initialisation happens can’t we just use old primary on restarted node as source for new replica to fasten things up? when you have 1000 shards with 30GB data each initializations of replicas on new nodes would be pain, big problem