Understanding Disk Shard Allocation

[corvintheus]

Hi, I have a cluster consists of 3 nodes with different sizes of disk. Now, whenever one of the nodes disk reached its high watermark. It seems that it still getting data and continuously going down. I am not sure what is wrong. My current disk size as of the moment is node1 have 100gb, node2 has 300gb and node3 has 1.5TB. Now I am wondering why its not going on node3 even node1 is already complaining about high watermarks? Thanks

cluster.routing.allocation.enable: all
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.flood_stage: 30gb
cluster.routing.allocation.disk.watermark.low: 100gb
cluster.routing.allocation.disk.watermark.high: 50gb

[spinscale (Alexander Reelsen)]

You night want to take a look at the cluster allocation explain API: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/cluster-allocation-explain.html