I am running a cluster in which all datanodes have their data dirs mounted on separate 6TB disks.
Today the cluster reached 84% disk usage and I was forced to delete a bunch of old indexes even though I still had almost 1TB free on each node.
Since the disks fils up equally the use case of having one node running out of disk is not relevant for me.
I am now considering increasing the defaults to something like 95%, but then I started wondering whether there is some reason behind having it set to 85%/90% by default?
Is there any special reason for those defaults and will it be fine if I increase to 95% or even higher?
My reasoning is that currently the cluster will anyway stop indexing when it reaches 85% (will stop allocate new shards on all datanodes), which will also happen if the cluster runs completely out of disk space. So in this case best to increase the defaults or disable disk usage checking at all. Does this makes sense?