I have 5 data nodes in my cluster, all of them split the data across two disks. There should be plenty of space for my data, but because elasticsearch allocates shards to disks based on shard count rather than size (and I understand you wouldn't know how big a shard is if it's still being written to), then I have situations where one disk gets really full, whilst the other one is only half full.
This triggers high watermark alerts and shards start bouncing between nodes.
My question is:
Is it possible to move shards between disks to more evenly balance the load?
One of the following would be great, in order of preference:
- Get elasticsearch to consider bouncing shards between disks before it starts offloading them to other nodes
- Have an endpoint that reallocates shards on-node between disks to even the load
- Have a tool that moves shards between disks
Does anything exist?