High disk watermark in elastcisearch

Elastic Stack Elasticsearch
1

Hello,
I am running ELK 1.5.2 n RHEL 6.
Everything was fine before I see below error in elasticsearch.log
Now there are no logs coming on Kibana.
My question are.
Are those logs going to some other location? Do I need to configure kibana to read from that index?
OR are those logs lost due to disk space issue?

Error stacktrace:

[2015-06-15 09:21:38,755][INFO ][cluster.routing.allocation.decider] [Mother Superior] high disk watermark exceeded on one or more nodes, rerouting shards
[2015-06-15 09:22:08,756][WARN ][cluster.routing.allocation.decider] [Mother Superior] high disk watermark [10%] exceeded on [3EnjH2mwSKurWpz0DDXfwg][Mother Superior] free: 416.1mb[6.9%], shards will be relocated away from this node

Please guide on this.

br,
Sunil.

2

It's likely the LS will stop processing events as ES cannot ingest anymore, so they won't be lost but you need to sort out your disk space issue.

3

Hello Mark,
Can you please elaborate the issue in detail?
What steps should I take to resolve this? I am not getting exactly what the problem is?

br,
Sunil.

4

As per the error - "high disk watermark exceeded....shards will be relocated away from this node" - assuming you have one node then it can never allocate new shards and data will stop flowing into ES.

Take a look at https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-allocation.html#disk

You can either decrease the watermark, add more disk to ES or delete data.