I am now trying to fully understand the cluster recovery process after encountered several times restart the cluster and the index data loss.
No question about the global and index meta data recovery. What confuse me is 'how ES make use of local index shard information when cluster recovery'.
From the source code, once meta recovery successfully an initializing RoutingTable (all shard unassigned) is created, then AllocationService will assign the shards to Live Nodes base on AllocationDecider. In my mind, each data node is supposed to report the local index shard information to MASTER node, master will rebalance dangling shard base on such information. But after searched the code, no such procedure happens.
Anyone can tell me how ES rebuild the routingTable during recovery phase?