Hello @DavidTurner,
There was another issue, a couple of the nodes were hitting the watermarks and others were close to it, so probably any shard movement could trigger another watermark that would then trigger another shard movement.
I removed some old data to make sure that no node in the warm tier would hit any watermark, and after sometime there are no more shard movements.
$ cat desired-balance.json | grep "node_is_desired" | grep -v relocating | grep false | wc -l
0