I was wondering, if there is possibility that elasticsearch automatically rebalances shards on nodes.
For example, when I have 2 or 3 or or more node cluster, and I restart single node, then after service restart, this node has all the indexes replica shards only. And to get some primary shards routed to this node, I'd have to manually do the reroute, for example, I restarted elasticnode2, then I'd have to manually reroute some shards for every index like this:
I can't even manually achieve this, since elasticnode2 has all the shards already allocated (replica shards), which get automatically allocated after node rejoin cluster (after service restart).
I get error when executing this cluster reroute command:
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[index_name1], node[K81sF6LQRP6VJaFJOqnYFg], [R], s[STARTED], a[id=mFB2EofwRdW_Ec9YkMQkOQ]]"
This is an interesting artefact of how Elasticsearch allocates shards: it will allocate the primaries first and then the replicas, one at a time, trying to maintain balance throughout the process, so when all the primaries are allocated and all of the replicas are unallocated then the balanced configuration is as you describe.
However it's not really necessary to maintain this balance, which is why it doesn't happen after a node restarts. Simpler just to leave the primaries where they are.
It shouldn't make a difference. All copies of a shard can service searches whether they be primary or replica.