I have been playing around with my ES cluster. Especially bringing down nodes randomly. I observe when I add the nodes back the primary and replica shards are rebalanced in some manner, I can't find the logic as in how that rebalance is made.
- One scenario, I observed out of 5, 4 primary shards were on a single node and all replicas spread evenly across rest of the nodes.
- Other scenario, when I added a node back, I saw primary shards were relocated, and not replica shards. I also observed that since these were primary shards, no other node/shard was promoted a primary. This would mean the writes would fail while this is happening since write can only happen in primary shard.
Whats the logic behind moving primary and replica shards across?