Is there much value in having different values for cluster.routing.allocation.disk.watermark.low/high configs?

(Bittu Sarkar) #1

Let's say the value of cluster.routing.allocation.disk.watermark.low and cluster.routing.allocation.disk.watermark.high are 85% and 90% respectively. Let's say a data node has 87% disk filled. Now, shard allocation will not be allowed on this node because it has exceeded the low watermark. If this node is restarted for any reason, all shards will remain unassigned due to the same reason. Elasticsearch will attempt to allocate some of the shards to other nodes and only when the disk usage is less than 85%, the remaining shards will initialize on this node. In a worse case, the time it takes to free up the 2% disk space is more than index.unassigned.node_left.delayed_timeout, shards will start allocating to other nodes thereby increasing recovery time. But, if we set the value of both the watermarks to 90% (effectively meaning there is no low disk watermark configuration), the situation would be as good as, if not better than having the low disk watermark set to 85%. Even after a data node restarts, all the shards would get initialized on the same node. Am I missing something here?