6.4.2 All nodes except one crosses watermark level

Elastic Stack Elasticsearch
1

I have 4 Data nodes in my cluster with size 2.5TB. Approx 250Gb data is added to the cluster every day. The lower disk watermark is 85%(default). We are facing an issue where replicas are not being assigned to any node.
The watermark level of each node is as follows:
Node1 : 86%
Node2 : 85%
Node3 : 89%
Node4 : 90%

All nodes have crossed their watermark level except Node2.

Why wasn't the load distributed across all nodes and all should have crossed their watermark level? Why only Node2 stopped accepting shards/data while others continued indexing data on their nodes?

2

I am confused by this question. All of the nodes have crossed the 85% watermark.

There are three disk-based watermark levels: low, high and flood_stage. Once a node crosses the low watermark it stops accepting new shards, but continues to write to any shards that it currently has. Once it crosses the high watermark it starts trying to relocate its shards elsewhere, but continues to write to any shards that it currently has while the relocation is ongoing. Once it crosses the flood_stage watermark it marks all its indices as read only and starts rejecting indexing traffic too. The documentation has more details: Disk-based shard allocation | Elasticsearch Guide [8.11] | Elastic

3

@DavidTurner Thanks :slight_smile:

What I meant was that Node2 never crossed 85% while others did.

But your answer explains the watermarking clearly and I was able to find the problem in my system.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.