I have 4 Data nodes in my cluster with size 2.5TB. Approx 250Gb data is added to the cluster every day. The lower disk watermark is 85%(default). We are facing an issue where replicas are not being assigned to any node.
The watermark level of each node is as follows:
Node1 : 86%
Node2 : 85%
Node3 : 89%
Node4 : 90%
All nodes have crossed their watermark level except Node2.
Why wasn't the load distributed across all nodes and all should have crossed their watermark level? Why only Node2 stopped accepting shards/data while others continued indexing data on their nodes?