Most challenging problems occur "when things have gone wrong". We have Warm Nodes that are oversharded and have network access storage mounted by mistake. The result is we have a good amount of data already on NAS, but the impact to performance now destabilizes the whole 35 node cluster daily.
We want to devise a way to move shards off the affected Warm Nodes in a gradual way while at the same time preventing any new shards from being moved to the affected Node. We don't want to dump all shards at once the way the allocation.exclude filter does--just prevent any new shards appearing on the target Node. We have dozens of policies and templates which are not organized / standardized so we're hoping for a cluster wide solution.
Is there a way to simply say "don't put any new shards on this node"?
Thanks!