Elasticsearch 6.8.6.
I'm wondering if someone can explain why after opening, unfreezing and freezing an index, some replica shards sometimes appear to be get recreated from primaries even though they already exist on disk. (Maybe they aren't being recreated, but that's what it seems like.)
Usually when I opened an index and watch Shard Activity in Kibana Monitoring I see "Recovery type: Existing Store" for every shard, all the shards are allocated quickly and the health goes Green. But sometimes I open an index and for some shards I see "Recovery type: Peer", "Source / Destination" shows that the shard is being copied from one node to another, elsewhere in Monitoring the shard is shown as Initializing, the unassigned shard count remains above 0, hence cluster health remains Yellow, until the recovery has completed. This is irritating because it can take a while with big shards and because I know that those shards exist on disk so why hasn't Elasticsearch used them.
I have witnessed this same behaviour after unfreezing an index and, most puzzlingly, after freezing an index. If I'm freezing an index which is Green, why would Elasticsearch decide that some of the shards have to be initialized by copying them from another node, making the index health Yellow for the duration, when all required shards obviously already exist?
In all these cases I can't find anything in the logs about why this is happening to shards in question. Asking the Explain API about a shard which appears to being recreated from primary returns
"explanation" : "the shard is in the process of initializing on node [whatever], wait until initialization has completed"
I know that sometimes Elasticsearch moves shards around in attempt to achieve it's idea of balance. In such cases the shards are shown as Relocating, not Initializing, and health remains Green. This makes me think that's what I describe above is Elasticsearch recreating the shards, not just relocating them. Also if it were the case that after opening, unfreezing or freezing an index Elasticsearch decided that some of the shards needed to be moved to achieve better balance, what I would expect to happen is that all shards are assigned using what's on disk, Green is quickly achieved, and then some shards are relocated. But maybe that's not how Elasticsearch works in that scenario.
Can anyone explain what's happening in the above scenarios or tell me where to look to maybe find out?