I have a cluster with 10 data nodes. some data nodes have multiple disks with different sizes (running on-prem, have to do with what i'm given).
During the night my main index got locked due to 'flood stage disk watermark'. My flood stage limit is configured to be 50gb, but when i checked the nodes in Kibana / 'GET /_cat/nodes' I couldn't see any node lacking disk space.
At the end it turned out that a single disk on one of my machines got filled (99%). This machine has two disks (600GB, 3.5TB) and the smaller one got filled. I think this behavior is a bit strange as i have no control over what shard goes to which disk, and the rest of the cluster had plenty of disk space. i would expect an ES node to know how to manage its disks and not lock the entire index/cluster when this could've been avoided.
My only option now, if i want to use this extra disk, is to have 2 ES instances on this machine (1 per disk), which is a waste of RAM and CPUs.
any thoughts/ ideas on that? have i missed something? should i open a bug?
(Running ES 6.8.6)
Thank you for your assistance.