It looks like the high disk watermark setting is compared to all available disk space on the machine and not only to the partition where ES writes its data. In our case we have a /data mount point and Elastic is running from /data/elasticsearch. The /data mount point exceeds the high water mark, but ES does not start relocating any shards from the server and we actually see the used space increasing further.
After looking at the code it looks like the free_in_bytes value of the fs stats is used and when comparing the watermark. On this specific server free_in_bytes appears to contain all available disk space on the machine (including / and /home). available_in_bytes on the other hand appears to contain the correct value of free disk space on the /data partition.
Can anyone confirm free_in_bytes is used for the watermarks and if it is expected to contain free disk space from irrelevant partitions?
And what is the exact difference between available_in_bytes and free_in_bytes?