Currently, I'm having three warm nodes, and ILM continuously ingesting data in warm nodes. Warm nodes is holding 85%, 80%, and 86% of the data, and data is continuously ingested.
So what happens when the warm tier gets filled up?
Elasticsearch has three disk-based watermark:
- low watermark, per default 85% of disk usage
- high watermarke, per default 90% of disk usage
- flood watermark, per default 95% of disk usage
When you hit the low watermark threshold, Elasticsearch will stop allocation new replica shards in the new nodes, new primary shards will still be created, but no replicas will be allocated in this node.
When you hit the high watermark threshold, Elasticsearch will try to relocate shards away from the node, sending them to other nodes, this applies to old and new shards.
And when you hit the flood stage watermark, Elasticsearch will set every index that has a shard on the node as read-only to prevente the node to filling up to 100%.
You can read more about these settings here.
Also, if you have large disks you may need to change those default values to not waste space.
When disk fills up Elsticsearch will stop allocating new shards to the nodes and try to relocate existing ones. This means migration of data from hot nodes can stop and those nodes fill up, in turn resulting in indices going read-only when the disks on the hot nodes fill up. I would recommend monitoring disk space and adjust data retention or cluster size as disks fill up.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.