I'm having an odd occurrence.
I had cluster running out of disk space. Added more nodes, and all but two of the nodes had room made on them.
2 nodes are still sitting above the high watermark.
I added another 3 nodes. and it is again moving the data off the other nodes, but not the 2 that are the most full.
It eventually moved it off 1 of those two nodes. but I still have one node sitting at 88% full disk
I can move shards off manually. Or exclude the node and force it to start moving. But seems like the cluster allocation should be doing it itself.
I think part of the problem is this.
"explanation": "there are too many copies of the shard allocated to nodes with attribute [zone], there are  total configured shard copies for this shard id and  total attribute values, expected the allocated shard count per attribute  to be less than or equal to the upper bound of the required number of shards per attribute "
I don't see where the upper bound is set however? zone awareness should be ensuring minimum per zone, not maximum from what I have read?
Any ideas on what is going on?