In a 3 node cluster (all nodes being master eligible), the master left the cluster and then rejoined after an hour. Soon, all primary and secondary shards(shard 9) became unassigned for an index.
I am also seeing the following logs from gateway.local: [index_name][9]: not allocating, number_of_allocated_shards_found [1], required_number [2]
As per _cluster/health API all primary and secondary shards for shard 9 are unassigned.
Why is there such a mismatch in the logs and the cluster health output? As per [gateway.local] logs, 1 shards is allocated, but the _custer/health API indicates that all primary and secondary shards are unassigned.