Suppose there is a two node cluster and indices have 1 replica. Initial shard allocation distributes primaries and replicas across both nodes.
node-1: [p0, p2, p4, p6, p8, r1, r3, r5, r7, r9] node-2: [p1, p3, p5, p7, p9, r0, r2, r4, r6, r8]
If one node goes down, all primaries get allocated on the remaining node (i.e. their replicas get promoted) --
node-1: [p0, p2, p4, p6, p8, p1, p3, p5, p7, p9]
When node-2 finally comes back up, it ends up with all replica shards, while all primaries are on node-1
node-1: [p0, p2, p4, p6, p8, p1, p3, p5, p7, p9] node-2: [r0, r2, r4, r6, r8, r1, r3, r5, r7, r9]
Is there any way to bring this cluster back to having both primaries and replicas across both nodes without adding a third node or temporarily reducing replica count?
We don't want all primaries on one node as primaries exert more stress on nodes in update scenarios that we commonly need. (https://github.com/elastic/elasticsearch/issues/41543)