I was doing an upgrade of a 5 node cluster 5.5 -> 5.6.4. 2 nodes are dataless. All indices have 1 replica.
After upgrading the first node, some indices went YELLOW and would stay so (this is in test environment, so I was watching closely and waiting for the cluster to become GREEN again). 59 shards stayed unassigned even many hours after the upgrade of the first node was done.
When running explain on the affected shards/indices, it correctly said that 3 nodes could hold those indices. For 2 of them it said "explanation": "target node version [5.5.0] is older than the source node version [5.6.4]". The third node already holds the primary shard, there the reason is "the shard cannot be allocated to the same node on which a copy of the shard already exists..."
So, this all sounds plausibile. And it is not really a problem, the cluster still works. Nevertheless I am a bit surprised because I would have expected the cluster to become GREEN again after the first node upgrade.
Is this really expected behavior? Or could I avoid this state by chosing a more clever upgrade path?
CU, Joe