High disk watermark [90%] exceeded on node,shards will be relocated away from this node

Elastic Stack Elasticsearch
1

HI All,

recently elasticsearch stops sending data to kibana as the disk space is exceeded.per day i can see 5 M documents are flowing from other server to Elasticsearch.Please find the error am getting

[2017-08-31T01:21:49,851][WARN ][o.e.c.r.a.DiskThresholdMonitor] [production] high disk watermark [90%] exceeded on [8klVIR6LQfOxAUcELG3wtA][production][/var/lib/elasticsearch/nodes/0] free: 20kb[2.8E-5%], shards will be relocated away from this node

could any one please suggest how can i avoid this issue?

I have tried to update the heap memory form xms2g to xms16g. still elasticsearch is not working.

Can anyone tell me where json documents actually stored?

Thanks in Advance !!!!!

Thanks
Shyamala

2

This is a disk space issue, you're low on disk space. You need to get a larger disk or free up space.

1 Like
3

right!

4

Thanks for the response!!!

If i delete few json documents..will elasticsearch run automatically?

5

Yes, i think. I haven't run into this issue though.

However, you may consider other other options as well - if you have indices, rolling daily, you may delete old indices.

You may refer to retiring data section in the definitive guide and snapshot and restore links below.

https://www.elastic.co/guide/en/elasticsearch/guide/current/retiring-data.html
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-snapshots.html

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.