Watermark floodstage not stopping usage of diskspace

Elastic Stack Elasticsearch
1

I have 1 node cluster where flood stage watermark is breached but it seems ES is still occupying data .Is this correct ?

PFB my logs :

[2020-05-30T06:18:44,774][WARN ][o.e.c.r.a.DiskThresholdMonitor] [elastic_226] flood stage disk watermark [95%] exceeew][elastic_226][/apps/efk/elasticsearch-7.6.0/data/nodes/0] free: 21.8mb[0%], all indices on this node will be marke
[2020-05-30T06:19:14,775][WARN ][o.e.c.r.a.DiskThresholdMonitor] [elastic_226] flood stage disk watermark [95%] exceeew][elastic_226][/apps/efk/elasticsearch-7.6.0/data/nodes/0] free: 18.4mb[0%], all indices on this node will be marke
[2020-05-30T06:19:44,776][WARN ][o.e.c.r.a.DiskThresholdMonitor] [elastic_226] flood stage disk watermark [95%] exceeew][elastic_226][/apps/efk/elasticsearch-7.6.0/data/nodes/0] free: 15mb[0%], all indices on this node will be marked
[2020-05-30T06:20:14,777][WARN ][o.e.c.r.a.DiskThresholdMonitor] [elastic_226] flood stage disk watermark [95%] exceeew][elastic_226][/apps/efk/elasticsearch-7.6.0/data/nodes/0] free: 11.6mb[0%], all indices on this node will be marke
[2020-05-30T06:20:44,779][WARN ][o.e.c.r.a.DiskThresholdMonitor] [elastic_226] flood stage disk watermark [95%] exceeew][elastic_226][/apps/efk/elasticsearch-7.6.0/data/nodes/0] free: 8.2mb[0%], all indices on this node will be marked
[2020-05-30T06:21:14,780][WARN ][o.e.c.r.a.DiskThresholdMonitor] [elastic_226] flood stage disk watermark [95%] exceeew][elastic_226][/apps/efk/elasticsearch-7.6.0/data/nodes/0] free: 4.8mb[0%], all indices on this node will be marked
[2020-05-30T06:21:44,781][WARN ][o.e.c.r.a.DiskThresholdMonitor] [elastic_226] flood stage disk watermark [95%] exceeew][elastic_226][/apps/efk/elasticsearch-7.6.0/data/nodes/0] free: 1.5mb[0%], all indices on this node will be mark

2

The flood stage causes Elasticsearch to make indices read-only and redistribute shards if it can. It does not delete any data.

3

So when u say indices are read only , does it mean new data and new indices wont be created ?
Also how do i limit ES from utilizing more disk space like 95% max .Isnt that what "flood stage watermark" supposed to do ?

4

Yes. No new indices will be created and no new data indexed.

5

So in this case how come disk availability is getting decreased continously even after watermark breach all the way to 0 % is my question

6

Do you have anything else writing there, e.g. logs?

7

No..it has only elastic search..but as I mentioned I do have only 1 node cluster...could that be the cause of the problem?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.